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1. 


INTRODUCTION 


1.1  Nature  of  the  Problem 

Current  estimates,  excluding  cancers  of  the  skin,  identify  breast  cancer  as  the  most 
commonly  diagnosed  malignancy  among  women  in  the  United  States. 

Telemammography  offers  the  potential  to  provide  improvements  in  efficiency  and 
effectiveness  over  the  current  mail-service  protocol  using  traditional  film-screen 
mammography  imaging.  The  sources  of  digital  mammography  can  be  either  digitized 
analog  film-screen  examinations  (50  micron  spot  size)  of  a  full  breast  digital 
mammography  unit  (40  micron  spot  size,  4k  x  6  k  x  2  Bytes/pixel).  It  is  likely  to  be 
three  years  before  full  breast  digital  mammography  units  will  become  widely  available. 
The  three  significant  technology  blockages  to  the  use  of  full  breast  digital  systems  in 
telemammography  are  the  following:  (a.)  the  transmission  of  large  size  records  (4k  x  6  k 
X  2  Bytes/  48  Mbytes/image;  8  images  for  a  total  file  of  384  Mbytes  per  screened 
examinations);  (b.)  the  grayscale  display  of  8  images,  each  48  Mbytes  wide;  and  (c.)  the 
archiving  strategy  of  these  large  image  files. 

(1).  Full  breast  digital  imaging  technology  promises  removal  of  the  limitations  of 
conventional  mammography,  due  in  part  to  the  detectors  (limited  latitude  and  contrast, 
lack  of  detection  efficiency,  and  film  granularity  noise) .  2.)  Two  significant  parameters 
of  digital  mammography  systems  are  the  spatial  resolution.  Using  screen  film 
mammography,  with  a  field  of  view  of  18  x  24  cm  or  24  x  30  cm,  the  limiting  resolution 

is  about  20  line  pairs  per  millimeter  (Ip/mm). 

This  would  require  a  spot  size  of  25  microns,  implying  a  matrix  size  of  9k  x  12k  and  a  2 


Byte  per  pixel  dynamic  range.  Due  to  the  efforts  of  scattering,  and  an  attenuation  factor 
between  7.5  to  75,  the  digital  array  size  is  4k  x  6k  x  2  Bytes  per  pixel.  Jaffe  and  co¬ 
workers  (2)  use  the  following  detector  design  characteristics:  (a.)  efficient  absorption  of 
the  radiation  beam;  (b.)  linear  response  over  a  wide  range  of  radiation  intensity;  (c.)  low 
noise;  (d.)  spatial  resolution  of  approximately  10  line  pairs  per  millimeter  (less  than  a  50 
micron  sampling),  (e.)  a  field  of  view  of  18  x  24-cm;  and  (4.)  acceptable  heat  loading  of 
the  x-ray  tube. 

Prototype  full  breast  digital  mammography  units  are  being  evaluated  at  selected  sites. 
We  are  evaluating  the  Bennett  Contour  mammography  machine.  This  digital 
mammography  unit  had  the  following  imaging  parameters  19  cm  x  25  cm  imaging  area; 
(b.)  13  line  pairs/mm  spatial  resolution;  (c.)  604  k  x  4.8  k  digital  array;  (d.)  single 
exposure;  21  fiber  optics  coupled;  (f)  14  bit  dynamic  range  per  pixel;  (g.)  image 
acquisition  rate  of  10  seconds;  (h.)  quantum  noise  limited;  (I.)  computer  is  Sun 
Sparcstation  20;  (j.)  networking  protocol  is  TCP/IP;  and  (k.)  software  is  combination  of 
UNIX,  X-Windows,  and  MOTIF-GUI. 

The  computer  system  is  based  upon  a  common  standard  bus  (SCSI,  S-bus,  and  VME- 
bus).  More  than  100  images,  each  48  Mbytes,  can  be  stored  on  a  single  disc.  The 
software  is  built  around  the  X-Window/MOTIF  graphical  user  interface.  This  system  is 
being  evaluated  so  that  the  full  potential  of  digital  mammography  can  be  determined  for 
the  early  detection  and  management  of  breast  cancer. 


1.2  Purpose  of  the  Present  Work 


The  research  hypothesis  being  tested  is  that  a  telemammography  system  can 
interpret  mammography  images  with  an  accuracy  level  sufficient  for  primary  diagnosis 
utilizing  a  film  digitizer  at  the  transmitting  site  (with  a  50-micron  pixel  size  for  spatial 
resolution  and  12  bit  pixel  range  for  contrast)  and  interactive  grayscale  display  monitors 
(2448  X  2560  x  8/12  bits)  at  the  receiving  site.  The  full  breast  digital  mammography 
(FBDM)  units  generate  a  4k  x  6k  x  2  byte  digital  array  for  each  image.  It  is  very  difficult 

to  design  display  protocols  for  FBDM  systems. 

A  successful  telemammography  system  will  provide  benefits  in  the  following 

areas: 

A.  PRIMARY  DIAGNOSIS.  Telemammography  offers  the  ability  to  provide 
mammographic  consultations  to  undeserved  and  remote  areas.  Achieving  the  image 
quality  required  of  a  telemammography  system  for  primary  diagnosis  will  enable  an 
outreach  program  to  enhance  a  region’s  breast  screening  programs  and  to  improve  patient 
care.  Expert  mammographic  interpretation  meeting  requirements  established  by  the 
Mammography  Quality  Standards  Act  can  be  monitored  to  localities  lacking  such 
expertise. 

B.  INTEGRATION  OF  MAMMOGRAPHY  GROUP  PRACTICE 
DISTRIBUTED  OVER  MULTIPLE  IMAGING  CENTERS.  As  the 

awareness  regarding  the  role  of  mammography  in  early  detection  of  breast  cancer 
increases,  so  does  the  need  for  more  accessibility  to  low  cost  screening  mammography. 
More  and  more  practices  are  responding  to  the  the  rapidly  growing  utilization  of 
mammography  by  opening  out  patient  clinics  to  imaging  practices.  Telemammography 


would  enable  a  group  with  a  limited  number  of  expert  mammographers  to  handle 
multiple  off-site  practices.  Additionally,  if  appropriate  for  the  practice,  the  radiologist 
could  supervise  screening  mammograms  off-site  and  determine  the  need  for  any 
additional  views  at  the  time  of  examination  instead  of  having  the  patient  return  at  a  later 
time.  Image  quality  could  also  be  supervised  off-site  via  telemammography.  Another 
advantage  of  this  system  is  that,  due  to  inefficiencies  of  scale,  mammography  costs 
would  be  lower  and  a  lower  fee  for  interpretation  could  be  maintained  without  the  need 
for  an  on-site  radiologist.  In  part,  this  would  be  related  to  alleviating  the  need  for  the 
physician  to  travel  to  and  from  the  various  satellite  screening  sites. 

C.  OVER  READING  OF  MAMMOGRAPHY  ISSUES.  There  is  increasing 
emphasis  on  the  interpretive  skills  of  radiologist’s  reading  mammograms  as  part  of  the 
quality  assurance  process  monitored  both  by  the  ACR  Mammography  Accreditation 
program  and  by  the  Food  and  Drug  Administration.  Residency  programs  are  offering 
more  time  in  mammography  rotations  now  compared  with  only  a  few  years  ago;  there 
have  been  formal  standardized  training  programs  for  radiology  residents  and 
mammographic  technologists  through  the  ACR-CDC  Cooperative  Agreement. 
Nonetheless,  the  impact  of  the  accreditation  guidelines  and  the  training  programs  will  not 
be  immediate,  and  there  remains  a  need  for  expert  mammographic  interpretation  in  many 
practices.  With  telemammography,  a  small  number  of  expert  mammmographers  could 
provide  consultation  services  or  second  readings  of  mammograms  for  a  larger  number  of 
general  radiologists,  and  improve  the  quality  of  care.  Additionally,  the  data  and  images 
for  patients  in  a  region  could  be  stored  and  utilized  for  the  development  of  a  regional 


mammography  database. 

D.  IMPROVED  CONSULTATION  WITH  SURGEON.  Primary  care 
physicians  and  surgeons  could  review  the  mammograms  on  their  patients  without  the 
need  for  '^signing  out' '  the  original  films.  On  a  broader  scale,  the  utilization  of 
telemammography  at  multiple  radiology  practices  in  a  referral  region  could  provide 
greatly  improved  access  to  a  patients’  prior  examination,  regardless  of  where  the  patient 
obtained  subsequent  mammograms.  The  importance  of  such  transmission  would  be 
multifocal:  original  films  would  not  need  to  be  mailed,  risking  their  loss,  the  cost  of 
making  copy  films  could  be  avoided:  and  the  facility  interpreting  the  current  study  would 
have  a  much  more  rapid  access  to  the  prior  exams,  thereby,  improving  the  accuracy  of 
diagnosis  expediting  the  diagnostic  evaluation  of  any  new  abnormalities,  limiting  the 
need  for  more  costly  diagnostic  consults  of  unchanged  findings  and  diminishing  the 
anxiety  of  the  patient  who  is  waiting  for  her  final  results. 


2. 


METHODS  OF  APPROACH 


2.1  Proposed  Tasks 

Three  tasks  are  required  in  support  of  evaluating  the  research  hypothesis. 

Task  1.  A  seleeted  set  of  analog  mammographic  films  have  been  collected  and 

digitized  using  a  laser  film  digitizer  set  at  50-micron  spot  size  and  a  12  bit  dynamic 
range. 

An  ROC  analysis  has  been  conducted  on  the  analog  mammographic  films  and  the 
digitized  films  are  to  be  displayed  on  grayscale  monitors  (2048  x  2560  x  8/12  bits). 

Task  2.  A  digital  communication  network  will  be  implemented  between  the 

Department  of  Radiology  Breast  Imaging  Center  in  the  Diagnostic  Center  for  Women 
(Primary  Care  Center  Building,  UVA)  and  the  off-campus  outpatient  Virginia 
Mammography  Center  (Northridge  facility,  UVA,  8  miles  from  the  campus).  A  laser  film 
digitizer,(5 0-micron  spot  size.  Model  150,  Lumisys  Inc.,  Sunnyvale,  CA)  and  computer 
workstation  (SUN,  SPARC  Model  40)  will  be  installed  at  Northridge.  Transmission  of 
the  digitized  mammographic  films  will  be  over  a  T-1  carrier  (1.544  Mbits/sec  signaling 
speed..  )  to  the  department’s  PACS  and  displayed  on  2048  x  2560  x  8/12  bit  grayscale 
monitors.  A  protocol  for  end-to-end  telemammographic  quality  control  will  be 
implemented. 

Task  3.  A  performance  evaluation  will  be  conducted  of  the  teleradiology  system 

using  the  metrics  of  response  time,  throughput.,  reliability,  and  clinical  acceptance. 

We  divided  the  above  three  tasks  into  the  following  aims: 

Aim  1  ■  Collection  of  an  adequate  retrospective  database  of  analog  mammographic 


film  images  and  patient  data  for  use  in  evaluating  a  telemammography  system. 


Aim  2.  Convert  the  collected  database  of  analog  mammographic  films  into  digital 

arrays  using  a  laser  film  digitizer  with  a  50-micron  pixel  spot  size  and  12  bits  per  pixel  of 
dynamic  range. 

Aim  3.  Conduct  an  ROC  analysis  of  the  retrospective  database  of  the  analog 

mammographic  images  and  the  digitized  arrays  displayed  on  the  2048  x  2560x  8/12  bit 
grayscale  monitors. 

Aim  4.  Implement  a  digital  transmission  service  between  the  Virginia 

Mammography  Center  at  Northridge  and  the  PACS  in  the  University  of  Virginia 
Department  of  Radiology  and  its  workstations  including  that  in  Diagnostic  Center  for 
Women. 

Aim  5.  Design,  implement,  and  evaluate  an  end-to-end  quality  control  program 

for  the  telemammography  system. 

Aim  6.  A  performance  evaluation  will  be  conducted  of  the  telemammography 

system  using  the  metrics  of  response  time,  throughput,  reliability,  and  clinical 
acceptance. 

2.2  Experimental  Methods  and  Results 
2. 1  Statement  of  Work. 

The  proposed  statement  of  work  for  the  contract  was  identified  by  the  year  and 
aim  as  follows.  We  present  these  tasks  and  aims,  commenting  on  our  current  progress  at 
the  completion  of  the  second  year  of  the  contract. 


Year  1. 


TASKl:  Aiml  COMPLETED  IN  YEAR  1 

•  Complete  collection  of  200  normal  and  200  biopsy-proven  malignant  analog 
mammographic  films  to  form  an  image  database  (6  months  to  complete). 

•  Collect  pathology  and  consultation  reports  for  the  400  images  in  the  database. 

•  Conduct  an  image  quality  control  protocol  on  the  image  database  to  insure 
correct  ground  truth  identification,  correct  diagnosis,  and  an  adequate  optical 
density  range  in  each  image. 

•  Conduct  a  review  of  the  identified  Regions  if  Interest  (ROIs)  to  insure  proper  I 
identification. 

During  year  1 ,  we  completed  the  collection  of  200  normal  and  200  abnormal  analog 
mammographic  films  to  form  our  image  database  (see  Appendix  analog 
mammographic  films  to  for  a  listing  coded  by  case  number).  Abnormal  cases  include 
benign  and  malignant  lesions,  with  pathology  serving  as  ground  truth.  We  have 
completed  the  collection  of  patient  data  and  have  added  the  patient’s  age  as  well  as 
demographic  data.  We  conducted  an  image  quality  control  protocol  on  the  analog  image 
database  to  insure  proper  ground  truth  identification,  correct  diagnosis,  and  the  proper 
optical  density  range  in  each  image.  All  cases  were  reviewed  and  lesions  were  analyzed 
and  classified  by  using  ACR  lexicon.  The  abnormal  cases  were  also  verified  for  presence 
of  only  one  lesion.  The  abnormals  selected  reflected  a  range  of  difficulty  in  lesion 
perception  and  analysis. 


Normal  mammograms  were  selected  as  normal  based  in  the  following:  (1.)  Initial 
consultation  reading  was  normal;  (2.)  review  of  images  showed  no  significant 
abnormality;  and  (3.)  follow  up  mammogram  at  least  24  months  later  showed  no  interval 
change. 

Mammographic  findings  of  intramammary  lymph  nodes,  classically  benign  calcification 
of  fat  necrosis,  dermal  calcifications  and  vascular  calcification  are  considered 
pathognomically  benign  and  could  be  present  on  “normal”  cases. 

Parenchymal  density  for  each  case  was  classified  on  a  scale  of  1  to  4  based  on  the 
ACR  lesion  (1  =  fatty;  2  =  scattered  fibroglandular  tissue;  3  =  heterogenously  dense;  and 
4  =  extremely  dense).  The  perendynal  density  of  normal  cases  was  matched  to 
abnormals.  There  were  an  approximately  equal  number  of  fatty  normals,  fatty  abnormals, 
etc. 

The  image  database  was  initially  collected  together  with  an  overlay  sheet  of  clear 
plastic  identifying  the  Regions  of  Interest  (ROI’s)  to  insure  proper  identification.  Our 
intent  was  to  digitize  this  ROI  and  use  it  for  display  on  the  grayscale  workstations.  We 
recognizes  a  bias  in  the  reader  response  because  of  the  use  of  ROTS.  That  is,  if  only  a 
Ik  X  Ik  ROI  is  displayed  to  each  reader,  a  bias  is  introduced  by  not  displaying  the  fiill 
image. 

TASKl:  Aim  2 

•  Digitized  the  400  analog  mammographic  images  with  a  50  micron  pixel  spot 
size  and  12  bits  per  pixel  of  dynamic  range. 

•  Conducted  a  review  of  the  digitized  images  using  the  grayscale  display 


workstations  (2048  x  2560  x  8/12  bits)  in  the  PACS  network. 

Each  of  the  film-screen  analog  images  were  digitized  to  50  micom  spot  size  using  a 
Lymis  Model  150  Laser  film  digitizer  (4k  x  4k  x  12  bits).  Then,  they  were  archived  onto 
4  mm  digital  archiving  data  tape.  All  400  mammography  examiniations  are  archived  into 
a  tape  library.  The  digitzed  film  screen  library  is  in  two  portions.  The  images  are 
archived  with  the  database  having  pointers  to  the  images  and  the  BIRAD  data. 

TASKl:  Aim  2 

•  Complete  digitization  of  the  collection  of  the  analog  mammographic  films  (two 
months  of  year  2,  began  in  year  1). 

THIS  TASK  HAS  BEEN  COMPLETED  IN  YEAR  2 
TASKl;  Aims 

Complete  ROC  analysis  of  mammography  analog  films  (requires  two  months  of 
year  2  to  complete  task  began  in  the  first  year).. 

THIS  TASK  HAS  BEEN  COMPLETED  IN  YEAR  2. 

The  400  cases  of  analog  image  were  interpreted  by  elder  readers  at  the  University  of 
Virginia  and  the  Medical  College  of  Virginia.  The  number  of  readers  was  expanded 
from  six  to  eleven.  Normal  and  abnormal  cases  were  randomized  out  the  films  were 
read  in  rounds  of  50  at  each  sitting. 

Image  interpretation  was  conducted  with  the  following  gradings  system: 

Masses  1.  (definitely  not  present);  2.  (probably  not  present);  3. 

(equivocal);  4.  (probably  present;  5.  (definitely  present). 


Microcalcification3  1.  (definitely  not  present);  2.  (probably  not  present);  3. 

(equivocal)  4.  (probably  present);  5.  (definitely  present). 


Dilated  lactiferous  ducts  1. :  2:  3;  4;  5;. 

Focal  areas  of  asymmetry  or  architectural  distortion  1;  2;  3;  4;  5. 

Diagnosis  of  image  1 .  (definitely  benign);  2  (probably  benign);  3.  (equivocal); 

4.  (probably  malignant)  5.  (definitely  malignant). 

We  have  completed  the  ROC  analysis  of  the  readers,  (appendice  II) 

In  year  two  we  have  also  completed  the  coding  of  all  cases  according  to  the  ACR  lesion 
with  description  and  pathologic  classification.  The  information  has  been  collected  in  the 
ACR  BIRAD  Program  Database  (appendice  III). 

Year  2. 

For  the  year  2  the  following  Tasks  and  Aims  were  to  be  accomplished. 

TASKl:  Aim  3 

•  Utilize  the  collected  digitized  image  data  set  to  perform  an  ROC  curve  analysis 
(requires  six  months)  utilizing  the  2048  x  2560  x  8/12  bit  grayscale  display  stations  in 
the  University  of  Virginia  PACS  by  six  readers. 

DUE  TO  THE  TECHNICAL  DIFFICULTIES  WITH  THE  GRAYSCALE 
DISPLAY  MONITORS,  THIS  TASK  IS  TO  BE  COMPLETED  IN  YEAR  3. 

•  TASK  2:  Aim  4 

•  Implement  the  T-1  connection  between  Northridge  facility  and  the  University  of 
Virginia  PACS  (three  months  of  the  Year  2). 


Test  network  for  end-to-end  fidelity. 


The  T-1  connection  between  Northridge  facility  and  the  University  of  Virginia  PACS  was 
to  be  installed  a  no  cost  to  the  contract.  This  effort  has  been  delayed  to  year  3  for  the 
following  reasons: 

1 .  The  UVA  PACS  high  resolution  display  stations  are  2k  x  2.5k  with  frame  buffers 
that  are  16  Mbytes  wide.  However,  the  boundary  of  all  acceptable  images  to  be  displayed 
has  been  set  to  22k  x  2.5k,  thereby  making  it  impossible  to  place  4k  x  4k  digitized  film- 
screen  mammograms  into  the  frame  buffer.  We  have  asked  E-SYSTEMS  to  modify  the 
software  but  thus  far  they  have  not  made  the  necessary  changes  so  we  can  display  the  50 
micron  digitized  mammograms. 

2.  By  digitizing  the  film-screen  mammograms  at  70  micron  spot  size  and  then 
thresholding  the  images  (removing  the  non-breast  tissue  portion  of  the  image),  they  are 
reduced  to  2k  x  2.5k.  However,,  such  a  reduction  prohibits  the  comparison  of  an  analog 
film-screen  ROC  analysis  to  image  that  of  a  70  micron  spot  size  image. 

3.  A  four  monitor  grayscale  display  system  is  being  developed  by  the  investigators 
to  enable  the  grayscale  display  of  4k  x  4k  images.  This  portion  of  the  study  will  be 
delayed  and  conducted  during  year  3  of  the  study. 

4.  An  ATM  connection  is  being  installed  from  the  MCV  Stoney  Point 
Mammography  office  to  MCV  Nelson  Clinic  for  transmitting  digitzed  screen-film 
mammograms.  Another  testbed  will  be  a  second  MCV  satellite  mamamography  office  in 
Blauster,  VA,  a  rural  site  approximately  70  miles  from  Richmond. 


TASK  2:  Aim  5 


•  Design  ,  establish,  and  test  an  end-to-end  quality  control  program  for  validating  a 
telemammography  system. 

•  Operate  the  telemammography  system  to  collect  data  for  evaluating  the  quality 
control  program. 

We  have  designed  and  validated  an  end-to-end  quality  control  program  using  phantoms. 
The  data  has  been  collected  intra-testbed  in  year  2.  In  year  3  we  plan  to  conduct  the 
test  using  an  inter-site  protocol.  The  first  test  will  utilize  the  ACTS  satellite  and  will 
begin  in  November  1996. 

For  years  3  and  4  the  following  Tasks  and  Aims  are  to  be  accomplished. 

YEAR  3 
TASK  1:  Aim  3 

•  Complete  the  ROC  analysis  of  digitized  mammographic  images  displayed  on 
2048  X  2560  x  8/12  bit  grayscale  display  stations  in  the  University  of  Virginia 
PACS. 

TASK  2:  Aim  5 

•  Implement  the  end-to-end  quality  control  program  for  evaluation  and  analysis. 

TASK  3:  Aim  6 

•  Implement  a  software  data  logger  program  which  will  record  events  on  the 
telemammography  system. 

•  Implement  the  performance  evaluation  using  the  metrics  of  response  time, 
throughput,  reliability  and  clinical  acceptance. 


3-  the  problem  of  displaying  full  breast  digital  mammography  examinations 

It  is  well  known  that  breast  cancer  is  the  most  common  form  of  cancer  in  American  women.  It  is  the  second 
leading  cause  of  cancer  related  deaths  among  women  only  surpassed  by  lung  cancer.  The  promising  increase  in  the 
rate  of  detected  breast  cancer  is  believed  to  be  partly  due  to  improved  screening  through  mammography 
examinations  (4).  Currently,  almost  all  mammography  is  performed  using  screen  film  systems  and  x-ray  units 
dedicated  to  performing  mammography  examinations.  There  are,  however,  several  limitations  in  the  use  of  screen- 
film  systems  for  mammography.  Screen-film  mammography  is  limited  in  detecting  cancers  in  patients  with 
radiodense  breast  tissue  (8).  These  women  make  up  about  40%  of  the  general  population  (9).  Digital 
mammography  reduces  the  limitations  of  conventional  screen-film  imaging  which  are  due  to  the  detector  (restricted 
latitude  and  contrast,  lack  of  detection  efficiency,  and  to  the  impact  of  film-granularity  noise)  and  to  the  image 
acquisition  (inefficiency  of  scatter  rejection)  (10).  Considerable  experience  with  digital  mammography  systems 
has  been  obtained  from  several  hundred  small  field,  spot  image  units  that  have  been  installed.  The  units  acquire 
small-field,  spot  images  during  needle  localization  or  core  biopsy  procedures.  These  units  have  demonstrated  the 
following  advantages:  (a)  shorter  procedure  times;  (b)  improved  image  quality  provided  by  the  targe  pixel  dynamic 
range  and  wide  linear  latitude  of  the  CCD  technology;  (c)  lower  overall  dose;  (d)  reduced  scattering  which 
improves  image  quality;  (d)  separation  of  the  x-ray  imaging  system  and  the  grayscale  display;  and  (e)  the  ability 
to  acquire,  transmit,  and  digitally  archive  the  images. 

The  clinical  acceptance  and  use  of  full-breast  digital  mammography  systems  now  depends  upon  developing 
display  protocols  that  a  radiologist  can  effectively  utilize.  The  three  parameters  used  in  specifying  a  digital  display 
are  the  pixel  array  size,  the  pixel  dynamic  range,  and  the  throughput  display  rate.  The  spatial  resolution 
requirements  for  digital  mammography  are  not  known  but  a  number  of  studies  have  led  to  the  conclusion  that  50-p 
pixel  sizes  are  a  reasonable  choice(lO).  This  implies  that  each  digital  mammographic  image  will  require  a  4k  x  4k 
pixel  display  array.  The  actual  dynamic  range  required  of  the  digital  mammography  image  is  also  not  known 
exactly  but  is  believed  to  be  between  12  and  14  bits  (10)  of  intensity  range.  The  throughput  display  rate  required 


is' believed  to  be  approximately  1  to  1.5  seconds  per  image  with  the  capability  of  displaying  at  least  four  images  at 
a  time.  Only  two  display  technologies  are  available  for  displaying  digital  mammography  images:  the  high 
resolution  laser  film  printer  (can  print  4k  x  6k  x  12  bit  images  with  optical  density  ranges  of  3.0  on  8x10  inch  size 
film)  and  the  grayscale  interactive  workstation  (a  single  monitor  can  display  2k  x  2.5  x  8  bit  images  from  a  32  M 
pixel  frame  buffer).  Studies  are  needed  to  determine  the  acceptable  display  protocols  for  the  clinical  display  of 
digital  mammography  images. 

The  advantages  of  the  high  resolution  laser  film  printer  over  a  workstation  are  the  following:  (a)  the  spatial 
resolution  of  the  laser  printed  digital  mammography  image  matches  the  acquired  digital  image  (4k  x  6k) ;  (b)  the 
size  of  the  laser  film  printed  image  matches  that  of  the  original  digital  mammography  image;  and  (c)  once  printed 
and  processed,  the  laser  film  printed  image  can  be  displayed  on  mammography  view  boxes  and  then  managed  in 
the  same  manner  as  standard  screen  film  images.  The  disadvantages  of  the  laser  film  printer  are  the  following:  (a) 
requires  20  seconds  per  image  for  exposing  the  latent  image  (prints  1  line  per  2.2  msec)  and  then  the  standard  90 
seconds  to  develop  the  film  before  clinical  review  of  each  image  is  possible;  (b)  the  optical  density  range  is  difficult 
to  match  operator  expectations  based  on  screen-film  examination  (have  to  develop  acceptable  look-up  tables);  and 
©  the  laser  film  printed  image  cannot  be  interactively  adjusted  for  window  and  level  settings. 

The  advantages  of  the  interactive  grayscale  workstation  are  the  following:  (a)  the  ability  to  interactively 
modify  the  display  image  throughout  the  12  bit  range  (window-level,  zoom  image  processing,  computer-aided 
diagnosis  algorithms);  (b)  the  use  of  multiple  displays  for  comparing  images  (current  and  previous  examinations); 
©  rapid  retrieval  and  display  from  the  archiving  storage;  and  (d)  design  and  use  of  individual  display  protocols. 
The  disadvantages  of  the  interactive  grayscale  workstation  for  digital  mammography  are  the  following:  (a)  it  is  only 
possible  to  view  a  2k  x  2.5k  portion  of  the  full  resolution  4k  x  4k  image;  (b)  sub-sampling  is  required  to  display  the 
full  size  digital  mammography  image;  (c)  multiple  monitors  are  required  for  each  interactive  workstation  to  display 
the  two  CC  views  and  two  MLO  views;  and  (d)  the  user  throughput  will  decrease  due  to  2k  x  2.5k  display  windows 


and  the  use  of  interactive  functions. 


3.1  HIGH  RESOLUTION  LASER  FILM  PRINTER 

None  of  the  current  full  breat  digital  mammography  units  can  disply  their  generated  images  at  full  resolution  (4k 
X  6k  x2  Bytes  per  pixel).  The  currently  available  high  resolution  laser  printer  can  print  4k  x  5200  lines  x  2  Bytes 
per  pixel.  This  printer  is  a  Kodak  printer  and  we  are  wrking  with  it  and  Kodak  to  determine  the  best  way  to  print 
4k  X  6k  X  2  Bytes  per  pixel. 

The  high  resolution  laser  filim  printer  requires  well-designed  look-up  tables  for  printing  digital  mammography 
images.  A  digital  mammography  unit  generates  12-bit  pixel  intensity  values;  the  laser  printer  accepts  12  bits  into 
its  memory  imit  but  prints  8  bits,  according  to  definitions  in  the  look-up  table.  New  look-up 
tables  can  be  installed  in  the  laser  imager.  The  investigators  for  this  project  has  had  extensive  experience  in 
developing  and  installing  look-up  tables.  The  look-up  tables  were  designed  for  use  with  a  laser  imager  with 
an  OD  range  of  0.2  to  3.0.  On  the  UVA  PACS  we  have  implemented  selectable  look-up  tables  for  a  3M 
Corp.  laser  film  printer  (8-bit  look-up  tables).  These  look-up  tables  have  proven  excellent  for  displaying 
anatomic  objects  in  the  printed  image.  The  standard  features  of  abnormal  mammograms  are  of  interest: 
masses,  microcalcifications,  architectural  distortion,  and  focal  asymmetries.  As  a  starting  point,  we  have 
adopted  16  graphs  similar  to  the  look-up  tables  used  in  the  FUJI  computed  radiography  system. 

During  the  third  year,  we  will  evaluate  the  curves  as  follows.  Two  experiments  will  be  conducted:  a 
contrast-detail  study  using  a  phantom  and  15  cases  from  the  digital  mammography  unit,  each  showing  one  of 
the  mammographic  features  of  interest.  An  evaluation  will  be  conducted  to  determine  the  best  look-up  table 
for  printing  digital  mammography  images  with  mammographic  features.  This  will  be  accomplished  by  having 
four  mammographers  rate  all  the  images,  using  the  scale;  1=  definitely  visible;  2=  probably  visible;  3  = 
equivocal;  4=  probably  not  visible;  and  5=  definitely  not  visible.  An  average  of  these  reader  scores  will  be 
calculated.  The  result  of  this  study  will  be  the  selection  of  the  optimum  look-up  tables  for  the  laser  imager. 

For  daily  QC  of  the  laser  imager,  we  will  employ  four  internally  generated  test  patterns.  These  will  be 
printed  and  developed  using  a  standard  film  processor  which  is  subject  to  the  daily  QC  dictated  by  MQSA. 
The  imager  calibration  test  pattern  has  17  bars  of  gray-level  densities,  in  increments  beginning  with  0.18  OD. 


"  A  densitometer  is  used  to  determine  if  the  change  in  density  from  one  bar  to  the  next  is  correct  and 
approximately  constant  for  a  linear  look-up  table.  The  attenuator  test  measures  32  positions  of  the  attenuator. 
The  universal  test  pattern  prints  resolution  bars  with  spacing  down  to  one  pixel.  The  flat  gray  test  pattern 
print  out  is  stored  as  a  record  of  the  laser  engine  performance. 

3.2  GRAYSCALE  DISPLAY  FUNCTIONS 

The  interactive  grayscale,  2-monitor,  workstation  provides  image  manipulation  and  enhancement  functions 
through  use  of  a  graphical  user  interface  (GUI).  The  following  performance  functions  are  already  implemented: 
(a)  worklist/  patient  list;  (b)  soft  button  using  icons;  (c)  image  selection  by  mouse-driven  cursor;  (d)  image 
rearrangement  and  display;  (e)  double  click  image  to  full  size;  (f)  next  exam;  (g)  image  enhancements;  (h)  window- 
level  setting;  (i)  automatic  histogram  equalization;  (j)  inverse  video;  (k)  zoom;  (1)  image  roam;  (m)  digital 
magnifying  glass;  (n)  rotation  and  flip;  (o)  undo  function;  (p)  system  working  message;  and  (q)  screen  saver. 

The  interactive  grayscale  user  display  functions  that  are  to  be  added  during  this  research  are  the  following: 

(a)  electronic  shutter;  (b)  image  data  compression  (wavelet  image  compression  at  50:1  and  a  screen  message  stating 
that  the  displayed  image  is  compressed);  (c)  nonlinear  look-up  tables,  similar  to  those  of  the  FUJI  look-up  tables 
(message  on  display  screen  stating  that  non-linear  look-up  tables  are  in  use);  (d)  DICOM  3.0  data  from  digital 
mammography  unit  to  be  displayed  in  a  screen  window  (radiation  exposure  parameters,  patient  ID  number,  patient 
name,  menstruation  history,  annotations,  additional  image  marking,  identified  follow-up  examinations,  and  BIRADS 
data);  and  (f)  the  design  and  implementation  of  10  display  protocols  to  be  evaluated.  An  example  is:  Monitor  1 
displays  a  current  exam  (craniocaudal  [CC],  left  and  right  breast,  mediolateral  oblique  [MLO],  left;  and  right  breast); 
Monitor  2  meanwhile  displays  either  previous  exams  if  available  (CC-L&R;  MLO-L&R)  or  previous  and  current 
left  CC;  previous  and  current  left  MLO;  etc.).  Two  significant  efforts  are  required  to  implement  acceptable  display 
protocols  for  a  digital  mammography  gray-scale  workstation:  (a)  development  and  evaluation  of  the  protocols;  and 

(b)  hardware  implementation. 

The  main  difficulty  for  grayscale  display  monitors  is  the  development  of  display  protocols  for  4k  x  6k  x  2  Bytes 


per  pixel  images.  We  will  trade  off  the  screen  display  of  2k  x  2k  images  and  of  the  4k  x  6k  full  breast  digital 
images. 


YEAR  4 


TASK  2:  Aim  5 

•  Evaluate  the  end-to-end  image  quality  control  protocol  for  the  teleradiology  system. 
TASKS:  Aim  6 

•  Evaluate  the  performance  evaluation  of  the  teleradiology  system . 

•  Continue  with. utilization  of  the  teleradiology  system  to  increase  the  statistical  power 

of  the  analysis. 

The  Archiving  Problem  For  Full  Breast  Digital  Mammography  Units 

We  have  digitized  400  film-screen  examinations,  200  normal  (left  and  right  breast,  CC 
and  MLO  screening  views.  All  of  the  400  cases  were  digitized  at  50 
micron  spot  size  and  50  of  the  library  were  also  digitized  at  70  micron  spot  size.  This 
digitized  mammography  library  is  archived  on  50  4mm  Data  Tape,  DDS-90,  3M  tapes 
(4mm  X  90  m,  295  feet).  We  have  a  database  for  pointing  to  the  patient  examinations. 
All  patient  digitized  mammograms  are  coded. 

The  library  is  a  huge  amount  of  dita  and  takes  many  hours  to  read  into  our  SUN 
workstation.  The  goal  for  third  year  is  to  place  this  data  on  CD-ROM’s  for  easier  access. 
The  images  are  to  be  archived  together  with  the  BIRAD  encoded  data  for  each  digitized 
examination. 

4.  Telemammography  Communication  Trials 

Two  communication  links  are  being  evaluated  for  telemammography.  The  Advanced 
Communications  Technology  Satellite  (ACTS)  will  be  used  between  the  Cleveland 
Clinic,  the  NASA  Lewis  research  Center,  and  the  University  of  Virginia.  A  Frame  Relay 
link  will  be  evaluated  between  The  Medical  College  of  Virginia,  Nelson  Clinic  and  the 


NASA 

C-91-11330 


MCV  satellite  practice  in  Blauster,  Virginia  (using  32:1  wavelet  compression).  Both  of 
these  telemammography  communication  links  require  a  method  for  displaying  the 
transmitted  images. 

4.1  Advanced  Communications  Technology  Satellite  (ACTS) 

The  investigators  have  an  in-linked  grant  from  NASA  to  investigate  the  application  of  the 
ACTS  for  teiemarrimography.  A  three  way  link  is  to  he  evaluated  between  UVA, 
Cleveland  Clinic,  and  NASA  Lewis  Research  Center.  The  primary  objective  is  to  utilize 
a  set  of  digitized  screen  film  mammogram  (50  micron  spot  size,  4  k  x  6k  x  2  Bytes  per 
image)  over  the  ACTS  satellite  link). 

The  ACTS  was  launched  in  September  1993  by  the  space  shuttle  Discovery  (STS-51). 
The  ACTS  is  in  a  geosynchronous  orbit  about  19,000  miles  above  the  equator  at  100 
degrees  W  longitude. 

The  ACTS  is  an  experimental  test-bed  designed  to  demonstrate  on  demand 
communication  links.  The  significant  features  of  the  ACTS  is  a  multi-beam  antenna, 
broadband  processor,  and  a  microwave  switch  matrix.  The  k-band  frequencies  are  used 
(uplink  27.5-30.0  Ghz  and  downlink  17.7-20.2  Ghz).  The  ka  band  (2.5  Ghz  bandwidth) 
is  being  used  for  reduced  antenna  size  and  for  high  capacity  datarates. 

The  antenna  being  used  is  the  ACTS  T1  VS  AT  with  a  1 .2  M  diameter.  This  antenna  is 
placed  on  a  roof  in  a  line-of-site  position  of  the  ACTS.  The  weight  is  minimal  so  that  it 
is  not  necessary  to  have  it  roof  mounted.  The  T1  VS  AT  is  an  active-phased  array  antenna 
consisting  of  a  multi-layered  microstrip,  EM-coupled  slot  and  dipole  monolithic 
microwave  integrated  circuit  (MMIC). 


Several  categories  of  digital  mammography  images  are  to  be  transmitted  across  the 
ACTS/MMIC  satellite  link.  They  will  include  the  following:  a.)  laser  digitized  analog 
film-screen  mammography  examinations  (4k  x  4k  x  2  Bytes  per  image).  This 
experiment  will  run  between  15  November  1996  to  15  January  1997. 

Due  to  the  problem  of  digital  mammography  display,  the  ACTS/MMIC  satellite  link 
experiment  will  be  a  round-robin  protocol  such  as:  a.)  UVA  to  ACTS  back  to  UVA  and 
printed  on  the  high  resolution  laser  film  printer  (4k  x  5200  x  12  bits);  b.)  Cleveland 
Clinic  to  NASA  Lewis  Research  Center  then  to  UVA  (three  legs).  UVA  has  the  high 
resolution  laser  film  printer  that  will  be  used  to  capture  the  resultant  transmitted  images. 
The  laser  film  printed  images  will  then  be  printed  and  graded  on  a  1  to  5  scale  with  1 
being  the  “best”.  The  film-screen  images  being  used  have  already  been  graded  and  will 
be  compared  to  the  transmitted  laser  printed  films.  Some  bias  will  be  acquired  but  use  of 
the  grayscale  workstation  will  not  enable  an  adequate  comparison. 

4.2  Frame  Relay  Communication. 

During  the  third  year  of  the  US  Army  contract,  we  will  install  a  Frame  Relay  connection 
between  The  Medical  College  of  Virginia  (MCV),  Nelson  Clinic,  and  Blauster 
mammography  practice.  The  cost  is  to  be  provided  by  UVA.  A  Frame  Relay  is  a  virtual 
leased  line,  reasonable  in  cost.  We  will  install  a  56  K  bps  Frame  Relay  link  with  a  32  to 
1  wavelet  image  compression  technique.  UVA  now  uses  this  type  of  communication  for 
its  Teleradiology  System. 

A  high  resolution  AGFA  laser  film  printer  (4k  x  5k  x  12  bits)  will  be  used  to  display  the 
transmitted  images.  A  two  monitor  AGFA  workstation  (2k  x  2k  x  12  bits)  has  been 


developed  and  will  be  demonstrated  at  the  InfoRAD  RSNA  '97  meeting  (December 
1996).  The  display  software  protocols  already  developed  will  be  utilized. 


5. 


Conclusions 


Implications  of  completed  work  at  the  end  of  the  second  year  of  activity  we  find 

several  important  results.  They  are  the  following: 

A.)  There  is  a  significant  need  to  develop  a  display  system  for  digital 
mammography  examinations. 

At  50  micron  spot  size  laser-film  digitized  analog  film-  screen  examinations,  the  digital 
array  size  is  4k  x  5k  x  12  bits.  The  full  breast  digital  mammography  systems  generate  a 
40  micron  pixel  spot  size,  the  result  being  4k  x  6k  x  2  Bytes  per  pixel. 

The  two  methods  for  displaying  any  digital  image,  are  with  a  laser  film  printer  and  an 
interactive  grayscale  display  workstation.  A  high  resolution  laser  film  printer  (4k  x  5200 
X  2  Bytes  per  pixel)  is  the  likely  choice  for  displaying  digital  mammography.  Our  50 
micron  digitized  analog  film  screen  mammography  examinations  (a  library  of  400 
examinations)  are  4k  x  5k  x  12  bits.  We  have  a  high  resolution  Kodak  laser  film  printer 
(4k  X  5200  lines  x  12  bits)  which  is  more  than  adequate  for  our  digitized  library. 
However,  our  FBDM  unit  produces  a  4.2k  x  6.4k  x  2  Bytes  digital  image.  We  are 
working  to  fit  this  FBDM  array  into  the  Kodak  printer  array.  Very  carefully  designed 
lookup  tables  have  been  implemented  and  are  being  tested. 

A  serious  difficulty  has  surfaced  in  the  use  of  our  UVA  PACS  two  monitor,  2k  x  2.5k  x 
8/12  bit  workstations.  The  frame  buffer  is  16  M  Pixels  but  is  portioned  such  that  only 
2k  X  2.5k  images  can  be  stored.  This  means  that  the  4k  x  4k  x  12  bit  50  micron  images 
can  not  be  displayed.  We  have  studied  all  possible  methods  but  ESYSTEMS  Software 
Staff  are  unwilling  to  modify  this  software.  Our  department  is  replacing  the  E-Systems 
PACS  and  will  use  two  monitor,  2k  x  2k  x  12  bit  workstations  in  the  near  future.  We 


have  acquired  such  a  workstation  (AGFA)  and  are  now  installing  the  software. 

B.)  Interactive  Grayscale  Workstation  Display  Protocols 

Acceptable  display  protocols  are  critical  in  using  interactive  gray-scale  monitors.  The 
acceptability  of  a  protocol  for  displaying  mammographic  images  may  be  judged  in  terms 
of  the  rapidity  with  which  a  user  can  accomplish  the  reading  tasks.  Image  processing  and 
management  steps  impact  the  throughput  rate  of  a  display  protocol,  as  do  the  demands  of 
mammographers  for  specific  organizations  of  images  on  the  screen.  As  an  example,  one 
possible  display  protocol  for  a  two-monitor  workstation  might  be  defined  as  follows. 
Monitor  1  displays  a  current  exam  (craniocaudal  (CC),  left  and  right  breasts;  mediolateral 
oblique  (MLO),  left  and  right  breasts).  Monitor  2  meanwhile  displays  either  previous 
exams  if  available  (CC-L&R;  MLO-L&R)  or  previous  and  current  left  CC;  previous  and 
current  left  MLO;  etc.  Data  from  the  radiology  and  the  hospital  information  systems  are 
displayed.  Pre-set  window  and  level  functions  could  aid  throughput,  as  could  prefetching 
(acquiring  the  patients  images  from  the  archive  file  and  storing  on  the  workstation,  an 
unacceptable  time  delay.  The  image  display  format  is  consistent  with  the  way  in  which 
they  will  be  reviewed  in  the  clinical  setting.  Mammograms  are  typically  viewed  as 
mirror  images,  and  if  a  lesion  is  identified  in  one  breast  the  two  views  of  that  breast  are 
reviewed. 

Examinations  will  be  stored  in  the  following  sequences: 

1.  Left  Mediolateral  Oblique  (MLO) 

Right  MLO 

2.  Left  Craniocaudal  (CC) 

Right  (CC) 


3. 


Left  MLO  and  Left  CC 


4.  Right  MLO  and  Right  CC 

Two  significant  efforts  are  required  to  implement  acceptable  display  protocols  for 
a  digital  mammography  gray-scale  workstation;  (a.)  development  and  evaluation  of  the 
protocols;  and  (b.)  hardware  implementation. 

First,  we  will  have  designed  several  plausible  display  protocols.  Second,  we  will 
evaluate  the  protocols  by  transferring  40  digitized  screen-film  mammography  cases 
from  the  PACS  to  an  optical  disk.  These  cases  will  be  equally  divided  among  masses, 
microcalifications,  architectural  distortions,  and  focal  asymmetries.  The  optical  disks 
will  have  the  images  preloaded  for  each  of  the  display  protocols  to  avoid  biasing  the 
protocol  evaluation  with  the  frustration  of  the  mammography  in  loading  a  prescribed 
sequence.  Third,  four  UVA  and  MCV  mammographers  will  evaluate  the  image  quality  in 
demonstrating  the  lesions  using  each  of  the  display  protocols.  A  reader  rating  scale  will 
be  used  for  each  case  (Example,  mass:l  =  definitely  acceptable;  2  =  probably  acceptable; 

3  =  equivocal;  4  =  probably  unacceptable;  and  4  =  definitely  unacceptable).  The  order  of 
each  question  will  be  randomized  as  well  as  the  cases.  The  reader  data  will  be  analyzed 
for  the  mean  score. 

The  times  of  initiation  and  completion  of  each  study  will  be  recorded  for  calculating  the 
throughput  times.  A  preferred  display  protocol  will  be  identified  on  the  basis  of  the  mean 
score  and  a  t-test. 

The  hardware  effort  is  to  implement  the  best  display  protocols,  as  evaluated  by 
the  mammography  readers,  onto  the  hardware  platform.  For  the  two-monitor  system. 


using  the  AGFA  system  as  a  test  bed,  we  will  incorporate  the  selected  display  protocol 
onto  a  DSP  board  using  toolkits  provided  by  HP.  HP  is  just  now  announcing  their  new 
accelerate  board;  we  expect  to  have  it  available  on-site  by  February  1996. 

We  are  currently  evaluating  a  set  of  display  protocols. 

C.)  Through-put  Performance 

Cost-benefit  analyses  for  digital  telemammography  lie  in  the  distant  future,  as 
they  will  need  to  reflect  currently  nonexistent  relationships  among  costs,  availability, 
efficacy,  and  quality-of-life  feats.  An  opportunity  to  analyze  initial  costs,  however, 

1  in  the  present,  created  by  the  availability  of  the  digital  mammography  environment 
described  in  this  application.  We  have  devised  a  cost  analysis  method  in  which,  for  any 
well-defined  system,  time  can  be  used  to  create  a  relationship  between  the  jobs 
accomplished  per  unit  of  time  ( the  throughput  rate)  and  resources  used  (costs)  to 
accomplish  those  jobs.  This  novel  strategy  should  be  applicable  to  any  mammography 
setting,  or  for  that  matter,  to  any  clinical  setting. 
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9/15/94  Through  8/30/96 
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5-25043  Dwyer 


Page  I 


Description 


Category 


CIr  Amount 


INFLOWS 

Bal  Fwd-5-25043  Dwyer 

9/15/94  Opening  Balance  [5-25043  Dwyer]  R  653,366.00 

TOTAL  TO  5-25043  Dwyer  653,366.00 

TOT/L  INFLOWS  653,366.00 


OUTFLOWS - 
20000: 


21100 


9/15/95 

Fac  Salary  to  date 

To  date 

20000:21100 

-27,835.35 

10/13/... 

Brookeman,  James  R 

Pay  period  end:  10/7 

20000:21100 

-342.64 

10/13/... 

Dwyer,  Samuel  J,  III 

Pay  period  end:  10/7 

20000:21100 

-1,540.00 

10/27/... 

Brookeman,  James 

Pay  period  end:  10/21 

20000:21100 

-342.64 

10/27/... 

Dwyer,  Samuel  J,  III 

Pay  period  end:  10/21 

20000:21100 

-1,540.00 

11/10/...  . 

Brookeman,  James 

1 1/4  pay  period 

20000:21100 

-342,64 

11/10/... 

Dwyer,  Samuel  J,  III 

1 1/4  pay  period 

20000:21100 

-1,540.00 

11/24/... 

Dwyer,  Samuel  J,  III 

11/18  pay  period 

20000:21100 

-1,540.00 

11/24/... 

Brookeman,  James 

11/18  pay  period 

20000:21100 

-342.64 

12/8/95 

Brookeman,  James 

12/02  pay  period 

20000:21100 

-338.75 

12/8/95 

Dwyer,  Samuel  J,  III 

12/02  pay  period 

20000:21100 

-1,522.39 

12/22/... 

Brookeman,  James  R 

12/16  pay  period 

20000:21100 

-343.33 

12/22/... 

Dwyer,  Samuel  J,  III 

12/16  pay  period 

20000:21100 

-1,538.53 

1/5/96 

Dwyer,  Samuel  J,  lit 

12/30  pay  period 

20000:21100 

-1,538.54 

1/5/96 

Brookeman,  James 

12/30  pay  period 

20000:21100 

-343.32 

1/19/96 

Dwyer,  Samuel  J,  III 

1/13  pay  period 

20000:21100 

-1,538.53 

1/19/96 

Brookeman,  James  R. 

1/13  pay  period 

20000:21100 

-343.33 

2/2/96 

Dwyer,  Samuel  J,  III 

1/27  pay  period 

20000:21100 

-1,538.53 

2/2/96 

Brookeman,  James 

1/27  pay  period 

20000:21100 

-343,33 

2/15/96 

Dwyer,  Samuel  J,  III 

2/10  pay  period 

20000:21100 

-1,538.53 

2/15/96 

Brookeman,  James 

2/10  pay  period 

20000:21100 

-343.33 

3/1/96 

Brookeman,  James 

2/24  pay  period 

20000:21100 

-343.33 

3/1/96 

Dwyer,  Samuel  J,  III 

2/24  pay  period 

20000:21100 

-1,538.53 

3/9/96 

Brookeman,  James 

3/09  pay  period 

20000:21100 

-343.33 

3/9/96 

Dwyer,  Samuel  J,  III 

3/09  pay  period 

20000:21100 

-1,538.53 

3/23/96 

Brookeman,  James 

3/23  pay  period 

20000:21100 

-343.33 

3/23/96 

Dwyer,  Samuel  J,  III 

3/23  pay  period 

20000:21100  . 

-1,538.53 

4/6/96 

Brookeman,  James 

4/06  pay  period 

20000:21100 

-343.33 

4/6/96 

Dwyer,  Samuel  J,  III 

4/06  pay  period 

20000:21100 

-1,538.53 

4/20/96 

Dwyer,  Samuel  J,  III 

4/20  pay  period 

20000:21100 

-1,538,53 

4/20/96 

Brookeman,  James 

4/20  pay  period 

20000:21100 

-343.33 

5/4/96 

Brookeman,  James 

5/04  pay  period 

20000:21100 

-343.33 

5/4/96 

Dwyer,  Samuel  J,  III 

5/04  pay  period 

20000:21100 

-1,538.53 

5/18/96 

Brookeman,  James 

5/18  pay  period 

20000:21100 

-343.33 

5/18/96 

Dwyer,  Samuel  J,  III 

5/18  pay  period 

20000:21100 

-1,538,53 

6/1/96 

Dwyer,  Samuel  J,  III 

6/0 1  pay  period 

20000:21100 

-1,538.53 

6/1/96 

Brookeman,  James 

6/01  pay  period 

20000:21100 

-343.33 

6/15/96 

Dwyer,  Samuel  J,  III 

6/15  pay  period 

20000:21100 

-1,538.53 

6/15/96 

Brookeman,  James 

6/15  pay  period 

20000:21100 

-343.33 

7/2/96 

Brookeman,  James 

6/29  pay  period 

20000:21100 

-343.33 

7/2/96 

Dwyer,  Samuel  J,  III 

6/29  pay  period 

20000:21100 

-1,538,53 

7/13/96 

Brookeman,  James 

7/13  pay  period 

20000:21100 

-343.32 

'  _  ‘  Register  Report  by  Category 

‘  '  9/15/94  Through  8/30/96 

10/2/96 

5-25043  Dwyer 

Date  Num  Description  Memo  Category  CIr  Amount 


7/13/96 

Dwyer,  Samuel  J,  III 

7/13  pay  period 

20000:21100 

-1,538.52 

8/2/96 

Dwyer,  Samuel  J.,  Ill 

Pay  ending  7/24 

20000:21100 

-1,538.53 

8/2/96 

Brookeman,  James 

Pay  ending  7/24 

20000:21100 

-343.33 

8/16/96 

Dwyer,  Samuel ).,  Ill 

Pay  ending  8/10 

20000:21100 

-1,538.53 

8/16/96 

Brookeman,  James 

Pay  ending  8/10 

20000:21100 

-343.33 

8/30/96 

Dwyer,  Samuel  J.,  Ill 

Pay  ending  8/24 

20000:21100 

-1,538.53 

8/30/96 

Brookeman,  James 

Pay  ending  8/24 

20000:21100 

-343.33 

TOTAL  21 100 

-72,982.37 

22100  : 

9/15/95 

Elias,  Beth  to  date 

To  date 

20000:22100 

-27,282.50 

11/1/95 

Elias,  Beth 

Oct  pay 

20000:22100 

-3,897.50 

11/30/... 

Elias,  Beth 

Nov  pay 

20000:22100 

-3,897.50 

12/22/... 

Elias,  Beth 

Dec  pay 

20000:22100 

-3,985.25 

1/31/96 

Elias,  Beth 

Jan  pay 

20000:22100 

-3,985.25 

2/29/96 

Elias,  Beth 

Feb  pay 

20000:22100 

-3,985.25 

3/29/96 

Elias,  Beth 

Mar  pay 

20000:22100 

-3,985.25 

4/30/96 

Elias,  Beth 

Apr  pay 

20000:22100 

-3,985.25 

5/31/96 

Elias,  Beth 

May  pay 

20000:22100 

-3,985.25 

6/30/96 

Elias,  Beth 

June  pay 

20000:22100 

-3,985.25 

7/31/96 

Elias,  Beth 

July  pay 

20000:22100 

-3,985.25 

8/30/96 

Elias,  Beth 

August  pay 

20000:22100 

-1,449.20 

TOTAL  22100 

-68,408.70 

23550 

9/15/95 

UVa Temps  to  date- 

To  date 

20000:23550 

-637.59 

2/29/96 

U.Va.  Temps 

Sam's  sec 

20000:23550 

-162.08 

TOTAL  23550 

-799.67 

TOTAL  20000 

-142,190.74 

30000: 


31000: 


31215 


9/15/95 

FB  to  date 

To  date 

30000:31000:31215 

-6,270.40 

10/31/... 

FB  Faculty  Oct 

Oct  FB 

30000:31000:31215 

-828.36 

11/30/... 

FB  Faculty  Nov 

FB  Fac  Nov 

30000:31000:31215 

-828.36 

12/31/... 

FB  Faculty  Dec 

FB  Fac  Dec 

30000:31000:31215 

-823.46 

1/31/96 

FB  Faculty  Jan 

FB  Fac  Jan 

30000:31000:31215 

-828.02 

2/29/96 

FB  Faculty  Feb 

FB  Feb  Fac 

30000:31000:31215 

-1,242.03 

3/31/96 

FB  Faculty  Mar 

Fac  FB  Mar 

30000:31000:31215 

-828.02 

4/30/96 

FB  Faculty  Apr 

FB  Fac  Sal  Apr 

30000:31000:31215 

-828.02 

5/31/96 

FB  Faculty  May 

FB  Fac  Sal  May 

30000:31000:31215 

-828.02 

6/30/96 

FB  Faculty  June 

FB  Fac  Sal  June 

30000:31000:31215 

-828.02 

7/31/96 

FB  Faculty  July 

FB  Fac  Sal  July 

30000:31000:31215 

-828.0! 

8/30/96 

FB  Faculty  Salary 

FB  Faculty  August 

30000:31000:31215 

-1,242.03 

TOTAL  31215  -16,202.75 
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'  ^  Register  Report  by  Category 

^  ^  9/15/94  Through  8/30/96 

10/2/96 

5-25043  Dwyer 


Date  Num  Description  Memo  Category  CIr  Amount 


31225 


9/15/95 

FB,  Class  to  date 

To  date 

30000:31000:31225 

-7,678.08 

10/31/... 

FB  Sal  Classified 

Oct  FB  Class 

30000:31000:31225 

-1,091.30 

1 1/30/... 

FB  Sal  Classified 

Nov  FB  Class 

30000:31000:31225 

-1,091.30 

12/31/... 

FB  Sal  Classified 

Dec  FB  Class 

30000:31000:31225 

-1,115.87 

1/31/96 

FB  Sal  Classified 

Jan  FB  Cl^sified 

30000:31000:31225 

-1,115.87 

2/29/96 

FB  Sal  Classified 

Feb  FB  Classified 

30000:31000:31225 

-1,115.87 

3/31/96 

FB  Sal  Classified 

Mar  FB  Classified 

30000:31000:31225 

-1,115.87 

4/30/96 

FB  Sal  Classified 

FB  Class  Apr 

30000:31000:31225 

-1,115.87 

5/31/96 

FB  Sal  Classified 

May  FB  Classified 

30000:31000:31225 

-1,115.87 

6/30/96 

FB  Sal  Classified 

June  FB  Classified 

30000:31000:31225 

-1,115.87 

7/31/96 

FB  Sal  Classified 

July  FB  Classified 

30000:31000:31225 

-1,115.87 

8/30/96 

FB  Sal  Classified 

FB  Classified  August 

30000:31000:31225 

-405.78 

TOTAL  31225 

-19,193.42 

.  TOTAL  3 1000 

-35,396.17 

34000: 


9/15/95 


9/15/95 


9/15/95 


34100 

Travel  -  to  date 
TOTAL  34100 
34350 


Veh  rental  to  date 
TOTAL  34350 
34500 


To  date 


To  date 


Lodging  &  other  to  date  To  date 
TOTAL  34500 
TOTAL  34000 
36000: 


30000:34000:34100 


30000:34000:34350 


30000:34000:34500 


-533.98 


-533.98 


-152.32 


-152.32 


-108.00 


-108.00 


-794.30 


36510 


9/15/95 

Other  re  product 

To  date 

30000:36000:36510 

-30.00 

1 1/30/... 

Other  re  product 

Nov  reproduction 

30000:36000:36510 

-12.50 

TOTAL  36510 

-42.50 

36960 

9/15/95 

Contractual  Services  to... 

To  date 

30000:36000:36960 

-1,000.00 

Page  3 


TOTAL  36960 


-1,000.00 


'  ^  Register  Report  by  Category 

^  9/15/94  Through  8/30/96 

10/2/96 

5-25043  Dwyer 


Date  Num  Description 


TOTAL  36000 
TOTAL  30000 
71100 


9/15/95 

Incjirects  to  date 

10/31/... 

Indirects 

11/30/... 

Indirects 

12/31/... 

Indirects 

1/31/96 

Indirects 

2/29/96 

Indirects 

3/31/96 

Indirects 

4/30/96 

Indirects 

5/31/96 

Indirects 

6/30/96 

Indirects 

7/31/96 

Indirects 

8/30/96 

Indirect  costs 

TOTAL  71 100 


TOTAL  OUTFLOWS 


OVERALL  TOTAL 


Category  CIr  Amount 


-1,042.50 


-37,232.97 


To  date 

71100 

-37,449.74 

Oct  indirects 

71100 

-4,982.87 

Nov  indirects 

71100 

-4,989.37 

Dec  indirects 

71100 

-5,027.14 

Jan  indirects 

71100 

-5,040.29 

Feb  indirects 

71 100 

-6,318.42 

Mar  indirects 

71100 

-5,040.29 

Apr  indirects 

71100 

-5,040.29 

May  indirects 

71100 

-5,040.29 

June  indirects 

71100 

-5,040.29 

July  indirects 

71100 

-4,943.34 

Indirect  Costs  -  August 

71100 

-4,458.72 

-93,371.05 


-272,794.76 


380,571.24 


Page  4 


TOTAL  INFLOWS 
TOTAL  OUTFLOWS 


324,945.00 

-139,788.58 


NET  TOTAL 


185,156.42 


APPENDIX  2 


CASE  # 

PARENCHYMAL 

GROUP 

FINDINGS 

DUGNOSIS 

3 

NORMAL 

2 

NORMAL 

3 

ABN 

CA,  AD,  FAD 

M 

3 

ABN 

CA 

B 

3 

ABN 

CA 

M 

4 

ABN 

CA 

B 

4 

ABN 

MASS 

B 

3 

ABN 

CA 

B 

1 

NORMAL 

010 

2 

NORMAL 

oil 

1 

NORMAL 

2 

ABN 

MASS 

M 

013 

1 

ABN 

AD,  MASS 

M 

014 

2 

ABN 

CA 

M 

3 

NORMAL 

016 

4 

NORMAL 

017 

1 

ABN 

MASS 

B 

2 

ABN 

CA 

M 

019 

2 

ABN 

CA 

M 

1 

ABN 

MASS 

M 

021 

.  2 

NORMAL 

022 

4 

NORMAL 

023 

2 

ABN 

FAD 

B 

024 

3 

ABN 

FAD 

B 

025 

2 

NORMAL 

026 

1 

NORMAL 

027 

4 

ABN 

CA 

B 

1 

NORMAL 

1 

ABN 

MASS 

B 

1 

ABN 

MASS 

B 

031 

3 

NORMAL 

032 

3 

ABN 

FAD 

B 

1 

ABN 

CA 

B 

034 

2 

NORMAL 

035 

4 

NORMAL 

036 

2 

ABN 

FAD 

B 

037 

3 

ABN 

CA 

B 

038 

3 

NORMAL 

039 

2 

NORMAL 

4 

ABN 

CA 

B 

041 

4 

ABN 

CA 

042 

2 

NORMAL 

043 

4 

NORMAL 

044 

2 

NORMAL 

045 

3 

ABN 

FAD 

046 

4 

NORMAL 

047 

3 

NORMAL 

04S 

2 

NORMAL 

049 

4 

NORMAL 

050 

2 

NORMAL 

051 

2 

ABN 

CA 

052 

■  3 

NORMAL 

053 

1 

NORMAL 

054 

1 

NORMAL 

055 

1 

ABN 

MASS 

056 

1 

ABN 

CA 

057 

1 

ABN 

MASS 

058 

2 

NORMAL 

059 

4 

NORMAL 

060 

1 

NORMAL 

061 

4 

ABN 

CA 

062 

2 

NORMAL 

063 

1 

NORMAL 

064 

3 

NORMAL 

065 

1 

ABN 

MASS 

066 

3 

NORMAL 

067 

4 

NORMAL 

068 

3 

ABN 

CA 

069 

3 

ABN 

MASS 

070 

3 

NORMAL 

071 

3 

ABN 

CA 

072 

3 

ABN 

CA 

073 

1 

NORMAL 

074 

2 

ABN 

CA 

075 

1 

NORMAL 

076 

4 

NORMAL 

077 

4 

NORMAL 

078 

3 

ABN 

MASS 

079 

2 

ABN 

MASS 

080 

2 

ABN 

MASS 

081 

3 

ABN 

CA 

082 

3 

ABN 

CA 

083 

1 

NORMAL 

084 

2 

NORMAL 

085 

1 

NORMAL 

S  S®  ®  w  w  cd  tfl 


086  3 

087  3 

088  1 
089  3 

090  3 

091  4 

092  3 

093  1 

094  1 

095  1 

096  4 

097  4 

098  1 

099  4 

100  3 

101  3 

102  3 

103  4 

104  1 

105  3 

106  2 

107  1 

108  2 

109  4 

110  1 

111  3 

112  2 

113  3 

114  3 

115  2 

116  2 

117  1 

118  1 

119  4 

120  1 

121  3 

122  4 

123  4 

124  1 

125  3 

126  1 

127  4 

128  2 

129  4 

130  1 


ABN 

MASS 

ABN 

CA 

ABN 

CA 

ABN 

FAD 

ABN 

MASS 

ABN 

CA 

NORMAL 

NORMAL 

ABN 

MASS,  CA 

ABN 

MASS 

NORMAL 

NORMAL 

NORMAL 

NORMAL 

ABN 

MASS 

NORMAL 

ABN 

AD 

ABN 

CA 

ABN 

MASS 

NORMAL 

ABN 

MASS 

NORMAL 

ABN 

AD 

ABN 

CA 

NORMAL 

ABN 

FAD 

NORMAL 

ABN 

FAD 

ABN 

CA 

NORMAL 

NORMAL 

NORMAL 

ABN 

MASS 

NORMAL 

NORMAL 

NORMAL 

NORMAL 

ABN 

CA 

ABN 

CA 

ABN 

CA 

ABN 

FAD 

NORMAL 

ABN 

CA 

ABN 

CA 

NORMAL 

131  3 

132  3 

133  1 

134  1 

135  2 

136  3 

137  2 

138  3 

139  2 

140  3 

141  1 

142  3 

143  4 

144  3 

145  4 

146  1 

147  1 

148  2 

149  4 

150  3 

151  1 

152  4 

153  1 

154  4 

155  3 

156  2 

157  3 

158  4 

159  4 

160  3 

161  2 

162  2 

163  3 

164  3 

165  2 

166  1 

167  4 

168  4 

169  4 

170  1 

171  3 

172  2 

173  3 

174  4 

175  4 


NORMAL 

ABN 

CA 

ABN 

CA 

ABN 

CA 

NORMAL 

NORMAL 

NORMAL 

ABN 

MASS 

ABN 

FAD 

NORMAL 

NORMAL 

ABN 

MASS 

ABN 

MASS 

ABN 

CA 

NORMAL 

ABN 

MASS 

NORMAL 

NORMAL 

NORMAL 

NORMAL 

ABN 

MASS 

NORMAL 

ABN 

MASS 

NORMAL 

NORMAL 

ABN 

NORMAL 

NORMAL 

NORMAL 

ABN 

ABN 

CA 

NORMAL 

ABN 

ABN 

AD 

NORMAL 

ABN 

FAD 

NORMAL 

ABN 

ABN 

CA 

ABN 

MASS 

NORMAL 

NORMAL 

NORMAL 

ABN 

ABN 

MASS 

mSoQ  SS  SSffl  S  m  S  «  Sm  mS  S  co  oa  S  aa  S 


176 

4 

NORMAL 

177 

1 

NORMAL 

178 

3 

ABN 

FCC 

B 

179 

2 

ABN 

CA 

B 

180 

4 

NORMAL 

181 

2 

ABN 

MASS 

B 

182 

4 

NORMAL 

183 

2 

NORMAL 

184 

3 

NORMAL 

185 

1 

NORMAL 

186 

1 

ABN 

CA 

B 

187 

3 

ABN 

MASS 

M 

188 

3 

ABN 

MASS 

M 

189 

4 

NORMAL 

190 

2 

NORMAL 

191 

3 

ABN 

CA 

B 

192 

4 

ABN 

CA 

M 

193 

3 

NORMAL 

194 

3 

ABN 

M 

195 

3 

ABN 

MASS 

M 

196 

2 

ABN 

MASS 

M 

197 

1 

NORMAL 

198 

2 

NORMAL 

199 

3 

NORMAL 

200 

3 

ABN 

MASS,  CA 

M 

201 

2 

NORMAL 

202 

■  1 

ABN 

M 

203 

2 

NORMAL 

204 

3 

NORMAL 

205 

2 

ABN 

M 

206 

2 

ABN 

B 

207 

3 

NORMAL 

208 

4 

ABN 

MASS 

B 

209 

3 

NORMAL 

210 

2 

ABN 

CA 

M 

211 

1 

NORMAL 

212 

4 

ABN 

M 

213 

4 

NORMAL 

214 

4 

ABN 

CA 

M 

215 

1 

NORMAL 

216 

4 

ABN 

CA 

M 

217 

2 

NORMAL 

218 

4 

NORMAL 

219 

3 

ABN 

CA 

M 

220 

1 

NORMAL 

221 

3 

NORMAL 

222 

3 

ABN 

223 

1 

NORMAL 

224 

1 

NORMAL 

225 

1 

NORMAL 

226 

1 

NORMAL 

227 

3 

NORMAL 

228 

3 

ABN 

CA 

229 

1 

NORMAL 

230 

3 

ABN 

FAD 

231 

3 

ABN 

MASS 

232 

4 

NORMAL 

233 

3 

ABN 

MASS 

234 

3 

ABN 

AD 

235 

1 

NORMAL 

236 

1 

NORMAL 

237 

4 

ABN 

CA 

238 

1 

ABN 

CA 

239 

2 

NORMAL 

240 

1 

NORMAL 

241 

2 

ABN 

242 

3 

NORMAL 

243 

1 

ABN 

MASS 

244 

2 

ABN 

MASS 

245 

3 

NORMAL 

246 

4 

ABN 

247 

4 

ABN 

CA 

248 

1 

ABN 

249 

2 

ABN 

CA 

250 

1 

ABN 

CA 

251 

3 

ABN 

MASS 

252 

3 

NORMAL 

253 

2 

ABN 

MASS 

254 

1 

ABN 

CA 

255 

4 

NORMAL 

256 

4 

NORMAL 

257 

2 

ABN 

CA 

258 

1 

NORMAL 

259 

2 

ABN 

MASS 

260 

4 

NORMAL 

261 

3 

ABN 

CA 

262 

2 

NORMAL 

263 

1 

ABN 

MASS 

264 

1 

ABN 

CA 

265 

4 

ABN 

CA 

M 


M 

B 

M 

M 

B 


^  tu  2;  w  w  ^  ^ 


266 

2 

NORMAL 

267 

1 

NORMAL 

268 

1 

ABN 

FAD 

269 

2 

NORMAL 

270 

2 

ABN 

CA 

271 

3 

ABN 

CA 

272 

4 

NORMAL 

273 

3 

ABN 

CA 

274 

3 

NORMAL 

275 

2 

ABN 

CA 

276 

4 

ABN 

CA 

277 

2 

NORMAL 

278 

4 

normal 

279 

3 

NORMAL 

280 

1 

NORMAL 

281 

3 

NORMAL 

282 

3 

ABN 

CA 

283 

4 

ABN 

CA 

284 

4 

ABN 

CA 

285 

3 

NORMAL 

286 

2 

NORMAL 

287 

1 

ABN 

MASS 

288 

4 

ABN 

CA 

289 

3 

NORMAL 

290 

1 

ABN 

MASS 

291 

2 

NORMAL 

292 

3 

NORMAL 

293 

3 

ABN 

CA 

294 

4 

NORMAL 

295 

3 

ABN 

CA 

296 

3 

ABN 

MASS 

297 

2 

NORMAL 

298 

4 

NORMAL 

299 

3 

ABN 

CA 

300 

1 

NORMAL 

301 

4 

ABN 

CA 

302 

4 

NORMAL 

303 

4 

NORMAL 

304 

3 

ABN 

CA 

305 

3 

ABN 

CA 

306 

2 

ABN 

MASS 

307 

4 

NORMAL 

308 

4 

ABN 

FAD 

309 

1 

ABN 

MASS 

310 

4 

NORMAL 

B 


M 

M 


M 

B 

B 

M 

M 


ss  s  s  s  mm 


311 

4 

ABN 

FAD 

M 

312 

2 

ABN 

MASS 

M 

313 

3 

NORMAL 

314 

3 

ABN 

CA 

M 

315 

4 

NORMAL 

316 

2 

ABN 

MASS 

M 

317 

2 

NORMAL 

318 

4 

NORMAL 

319 

3 

NORMAL 

320 

3 

ABN 

CA 

B 

321 

4 

NORMAL 

322 

2 

ABN 

CA 

M 

323 

4 

ABN 

CA 

M 

324 

4 

ABN 

CA 

B 

325 

2 

ABN 

FAD 

B 

326 

3 

NORMAL 

327 

2 

NORMAL 

328 

3 

ABN 

CA 

B 

329 

2 

NORMAL 

330 

2 

ABN 

MASS 

B 

331 

2 

NORMAL 

332 

1 

NORMAL 

333 

4 

ABN 

MASS 

M 

334 

2 

ABN 

FAD 

M 

335 

1 

ABN 

MASS 

M 

336 

2 

ABN 

MASS 

M 

337 

'  2 

ABN 

CA 

B 

338 

2 

NORMAL 

339 

3 

NORMAL 

340 

2 

NORMAL 

341 

3 

ABN 

CA 

M 

342 

3 

NORMAL 

343 

3 

NORMAL 

344 

2 

NORMAL 

345 

2 

NORMAL 

346 

3 

ABN 

CA 

M 

347 

2 

NORMAL 

348 

2 

ABN 

MASS 

B 

349 

2 

NORMAL 

350 

2 

NORMAL 

KEY 


PARENCHYMAL  DENSITY: 

1=FATTY 

2=SCATTERED  FIBROGLANDULAR  TISSUE 
3=HETEROGENEOUSLY  DENSE 
4=  EXTREMELY  DENSE 


GROUP: 

NORMAL=NORMAL 
ABN  =  ABNORMAL 

FINDINGS: 

MASS=MASS 

FAD  =FOCAL  ASYMMETRIC  DENSITY 
AD  =ARCHITECTURAL  DISTORTION 
CA  =CALCIFICATIONS 

DIAGNOSIS: 

M=MALIGNANT 

B=BENIGN 


READERS  RESPONSES  TO  CASE  84 


MASS 

CALCIFICATION 

FAD/ AD 

DIAGNOSIS 

1 

5 

1 

3 

2 

5 

1 

2 

2 

2 

1 

1 

1 

5 

1 

4 

1 

1 

2 

2 

1 

2  • 

1 

2 

1 

5' 

1 

4 

1 

5 

1 

3 

READER  # 

6 

5 

4 

3 

2 

7 

10 

11 


KEY  TO  FINDINGS: 
1=DEFINITELY  NOT  PRESENT 
2=PROBABLY  NOT  PRESENT 
3-EQUIVOCAL 
4=PROBABLY  PRESENT 


KEY  TO  DIAGNOSIS 

l=DEFrNITELY  BENIGN 
2=PROBABLY  BENIGN 
3=EQUIVOCAL 
4=PROBABLY  MALIGNANT 


5=DEFINITELY  PRESENT 


5=DEFINITLEY  MALIGNANT 


READERS  SPECIFIC  RESPONSES 


READER  6 


TRUE  NORMALS 

(149) 

TRUE  BENIGN 

(66) 

TRUE  MALIGNANT 
(84) 

DEFINITELY  BENIGN 

010 

01 

00 

PROBABLY  BENIGN 

121 

17 

14 

EQUIVOCAL 

017 

41 

30 

PROBABLY  MALIGNANT 

001 

07 

25 

DEFINITELY  MALIGNANT 

000 

00 

15 

READER  2 

TRUE  NORMALS 
(149) 

TRUE  BENIGN 
(66) 

TRUE  MALIGNANT 
(84) 

DEFINITELY  BENIGN 

28 

02 

01 

PROBABLY  BENIGN 

62 

17 

06 

EQUIVOCAL 

59 

37 

41 

PROBABLY  MALIGNANT 

00 

10 

23 

DEFINITELY  MALIGNANT 

00 

00 

13 

CLINICAL  HISTORY  SHEET 


MAMMOGRAPHY  CLINICAL  HISTORY  SHEET 

HISTORY  NO.  :  _ _  DATE  OF  SERVICE: 

NAME :  LAST 

ADDRESS :_ _ 

SSN: _  DATE  OF  BIRTH: _ 

HOME  PHONE  NUMBER:  ( _ ) _ - _ ^WORK  PHONE  :  ( _ ) 


IS  THIS  YOUR  FIRST  MAMMOGRAM?  Y  N 

IF  NO,  WHERE  WERE  YOUR  OLD  FILMS  DONE? 

WHEN  WAS  YOUR  LAST  MAMMOGRAM? 

HOW  OLD  WERE  YOU  WHEN  YOUR  PERIOD  STARTED? 

WHAT  IS  THE  DATE  OF  YOUR  LAST  PERIOD? 

HAVE  YOU  EVER  HAD  A  HYSTERECTOMY?  Y  N 

DID  THEY  REMOVE  YOUR  OVARIES?  Y  N 

HOW  MANY  TIMES  HAVE  YOU  BEEN  PREGNANT? 

HOW  MANY  CHILDREN  DO  YOU  HAVE? 

HOW  OLD  WERE  YOU  WHEN  YOUR  FIRST  CHILD  WAS  BORN?  _ 

DO  YOU  TAKE  BIRTH  CONTROL  PILLS?  _ 

HAVE  YOU  EVER  HAD  CANCER?  Y  N 

IF  YES,  WHAT  KIND  OF  CANCER? 

HAVE  ANY  OF  YOUR  FAMILY  MEMBERS  HAD  BREAST  CANCER? 

MOTHER _  SISTER _  AUNT  GRANDMOTHER  OTHER _ 

GIVE  AGE  AT  DIAGNOSIS:  _ _ 

DO  YOU  HAVE  BREAST  IMPLANTS?  Y  N 

IF  YES,  WHAT  KIND  OF  IMPLANTS? 

DO  YOU  TAKE  HORMONES?  Y  N 

WHAT  KIND  OF  HORMONES?  ESTROGEN  TAMOXIFIN 

PROGESTERONE  OTHER 

AT  WHAT  AGE  DID  YOU  BEGIN  TAKING  HORMONES? 


FIRST _ M.I. 

/ 

_  ZJIP: 


HAVE  YOU  EVER  HAD  BREAST  SURGERY?  Y  N 

IF  YES,  WHEN  AND  WHICH  BREAST? 

WHAT  WERE  THE  RESULTS? 


HAVE  YOU  EVER  HAD  RADIATION  THERAPY?  Y  N 

IF  YES,  WHICH  BREAST  AND  IN  WHAT  YEAR? 


HAVE  YOU  EVER  HAD  A  BREAST  REMOVED? 

IF  YES,  WHICH  BREAST? 

Y 

N 

HAVE  YOU  FOUND  ANY  NEW  LUMPS  IN  YOUR  BREAST? 
IF  YES,  WHICH  BREAST? 

Y 

N 

HOW  LONG  HAVE  YOU  HAD  THE  LUMP? 

HAS  THE  LUMP  CHANGED? 

DO  YOU  HAVE  ANY  OTHER  NEW  BREAST  PROBLEMS?  Y  N 

IF  YES,  PLEASE  DESCRIBE: 

WHEN  DID  THE  PROBLEM  START? 


MAMMOGRAPHIC  FINDINGS 


Mel§View” 

Finding  check-off 
sheets 


Paiicni  ID; 


Patient  Name: 


Examirtation  Date: 


□  Prior  study  dates  compared:  /  /  /  /  /  / 


Findinj*  _ _  of 


□Negative  exam 

Tissue  Density 

Recommendation 

□  Mammogram 

Q  Almost  entirely  fat 

□  Normal  interval  screening 

□  Ultrasound 

□  Scattered  fibroglandular 

in _ months  or  by  age _ 

Q  Duaography 

densities 

□  Any  dedsion  to  biopsy  should  be 

Q  Heterogeneously  dense 

based  on  clinical  assessment 

□  Extremely  dense 

Initials: 

Q Non-Negative  Finding 

□  Finding  correlates  to  clinical  exam  finding  in  QL  QR  UB  breast(s)  at  _ _ (location) 


_  • 

u  FoIIow-up 

□  FoIIow-up  of  prior  finding 

□  Change 

□  Decrease  in  number 

3  Follow-up  of  procedure 

in  UL  QR  Qb  breasi(s) 

□  No  significant  changes 

calcifications 

□  Lumpectomy 

at 

□  Increase  in  size 

□  Less  defined 

□  Excisional  biopsy 

(location). 

□  Decrease  in  size 

□  More  defined 

Q  Mastectomy 

Q  Increase  in  number  of 

□  Completely  removed 

□  Needle  biopsy 

calcifications 

□  Partially  removed 

Q  Radiation  Therapy 

Finding  Side.  □  Left  □  Right  J  Both  □  Multiple  similar  findings:  Approximate  number:. 

Mammogram 


3  Not  seen  on  mammogram 

Tissue  Density  (choose  one) 
3  Almost  entirely  fat 
3  Scattered  fibroglandular 
densities 

3  Heterogeneously  dense 
3  Extremely  dense 


Mass  Shape  (choose  one) 

□  Round 

□  Oval 

□  Lobular 

□  Irregular 

□  Architeaural  distonion 
Q  Tubular  density/solitary 

dilated  dua 

□  Intramammary  lymph  node 

□  Asymmetric  breast  tissue 

□  Focal  asymmetric  density 

Margins  (choose  one) 

□  Grcumscribcd 

□  MicroIobuJaied 

□  Obscured 

□  Indistina 
3  Spicufated 

Density  (choose  one) 

□  High  density 

□  Low  density 

□  Isodense 

□  Far  containing 


Gilcifications 

□  Skin 

□  Vascular 

□  Coarse 

□  Large  rod-like 

□  Large  round 

□  Eggshell  or  rim 
Q  Milk  of  calcium 

□  Dystrophic 

□  Punctate 

□  Amorphous  or  indistina 

□  Heterogeneous  or  pleomorphic 
Q  Fine  and/or  branching 

□  Spherical  or  lucent-centcred 

□  Suture 

Distribudon  (choose  one) 

Q  Grouped  or  clustered 

□  Segmental 

□  Regional 

□  Linear 

Q  Diffuse/scaitered 


Other  findings 

□  Nipple  retranion 

□  Skin  thickening 

□  Trabecular  thickening 

□  Skin  lesion 

□  Axilliary  adenopathy 

□  Skin  reiraaion 

□  Architeaural  distortion 

□  Hematoma. 

□  Post  surgical  scar 

Implant  Findings 

□  Asymmetric  implant 
Q  (^Idfied  implant 

□  Distoned  implant 

□  Fibrosed  implant 

□  Herniated  implant 

□  Ruptured  implant 

□  Free  silicone 

□  Capsular  contraaion 


READER 

02 

03 

04 
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ROC  RESULTS  OF  READERS  FOR  ANALOG  IMAGES 


R  0  C  F  I  T  (JUNE  1993  VERSION)  : 

MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMAI 


NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  59. 


RESPONSE  DATA: 
CATEGORY 

ACTUALLY  NEGATIVE 
ACTUALLY  POSITIVE 


1  2 

CASES  44.  44. 

CASES  3 .  3 . 


3  4  5 

53.  2.  6. 

8.  10.  35. 


OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0403  0.0537  0.4094  0.7047  1.0000 
TPF:  0.0000  0.5932  0.7627  0.8983  0.9492  1.0000 


INITIAL  VALUES  OF  PARAMETERS: 

A=  1.3744  B=  0.5371 

Z(K)=  -0.5376  0.2287  1.6104  1.7480 

LOGL=  -271.0403 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AE 


PROCEDURE  CONVERGES  AFTER  5  ITERATIONS-. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.4609  B=  0.6171 

Z(K)=  -0.5266  0.2289  1.4998  1.9286 

LOGL=  -264.3502 

CHI-SQUARE  GOODNESS  OF. FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AF 


VARIANCE-COVARIANCE  MATRIX: 


A 

0,0525 

0.0197 

0.0050 

0.0052 

0.0045 

0.0009 

B 

0.0197 

0.0173 

0.0020 

0.0011 

-0.0052 

-0.0117 

Z( 

1) 

0.0050 

0.0020 

0.0116 

0.0062 

0.0030 

0.0021 

Z( 

2) 

0.0052 

0,0011 

0.0062 

0.0105 

0.0055 

0.0047 

Z( 

3) 

0.0045 

-0.0052 

0.0030 

0.0055 

0.0234 

0.0231 

Z( 

4) 

0.0009 

-0.0117 

0.0021 

0.0047 

0.0231 

0.0397 

CORRELATION  MATRIX: 


A 

1.0000. 

0.6523 

0.2014 

0.2210 

0.1284 

0.0187, 

B 

0.6523 

1.0000 

0.1390 

0.0823 

-0.2592 

-0.4476 

Z( 

1) 

0.2014 

0.1390 

1.0000 

0.5645 

0.1807 

0.0962 

Z( 

2) 

0.2210 

0.0823 

0.5645 

1.0000 

0.3543 

0.2323 

Z( 

3) 

0.1284 

-0.2592 

0.1807 

0.3543 

1.0000 

0.7586 

Z( 

4) 

0.0187 

-0.4476 

0.0962 

0.2323 

0.7586 

1.0000 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 


TPF 


(LOWER  BOUND,  UPPER  BOUND) 


0.005 

0.4487 

( 

0.010 

0.5100 

( 

0.020 

0.5766 

( 

0.030 

.  0.6179 

( 

0.040 

0.6481 

( 

0.050 

0.6721 

( 

0.060 

0.6919 

( 

0.070 

0.7088 

( 

0.080 

0.7236 

( 

0.090 

0.7367 

( 

0.100 

0.7485 

( 

0.110 

0.7592 

( 

0.120 

0.7690 

( 

0.130 

0.7781 

( 

0.140 

0.7865 

( 

0.150 

0.7943 

( 

0.200 

0.8268 

( 

0.250 

0.8520 

( 

0.300 

0.8723 

( 

0.400 

0.9040 

( 

0.500 

0.9280 

( 

0.600 

0.9471 

( 

0.700 

0.9628 

( 

0.800 

0.9762 

( 

0.900 

0.9878 

( 

0.950 

0.9934 

( 

0.2635  ,  0.6460  ) 
0.3324  ,  0.6855  ) 
0.4124  ,  0.7283  ) 
0.4638  ,  0.7551  ) 
0.5019  ,  0.7751  ) 
0.5322  ,  0.7911  ) 
0.5573  ,  0.8046  ) 
0.5787  ,  0.8163  ) 
0.5973  ,  0.8266  ) 
0.6137  ,  0.8359  ) 
0.6284  ,  0.8443  ) 
0.6416  ,  0.8520  ) 
0.6537  ,  0.8591  ) 
0.6647  ,  0.8656  ) 
0.6749  ,  0.8718  ) 
0.6844  ,  0.8775  ) 
0.7234  ,  0.9015  ) 
0.7531  ,  ,  0.9200  ) 
0.7771  ,  0.9348  ) 
0.8147  ,  0.9568  ) 
0.8442  ,  0.9719  ) 
0.8693  ,  0.9826  ) 
0.8921  ,  0.9901  ) 
0.9142  ,  0.9952  ) 
0.9384  ,  0.9985  ) 
0.9538  ,  0.9995  ) 


ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT  LOWER  BOUND  UPPER  BOUND 

(  FPF  ,  TPF  )  (  FPF  ,  TPF  )  {  FPF  ,  TPF  ) 

(0.0269,  0.6067)  (0.0102,  0.5118)  (0.0620,  0.6956) 

(0.0668,  0.7038)  (0.0360,  0.6369)  (0.1151,  0.7644) 

(0.4095,  0.9065)  (0.3338,  0.8841)  (0.4887,  0.9256) 

(0.7008,  0.9629)  (0.6238,  0.9511)  (0.7696,  0.9723) 

1  R  0  C  F  I  T  (JUNE  1993  VERSION)  : 

MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  COLLECTED  IN  5  CATEGORIES 
WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMAL 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  82. 

RESPONSE  DATA: 

CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 


1  2  3  4  5 

127.  14.  5.  0.  3. 

12.  6.  4.  11.  49. 


OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0201  0.0201  0.0537  0.1477  1.0000 


r 


.TPF:  0.0000  0.5976  0.7317  0.7805  0.8537  1.0000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.8080  B-  0.6820 

Z(K)-  1.0466  1.6104  1.9514  2.0514 

LOGL=  -191.922.6 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AR 


PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.8169  B=  0.6938 

Z(K)=  1.0526  1.5531  1.8560  2.2459 

LOGL=  -184.4330 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AF 


VARIANCE -COVARIANCE  MATRIX: 


A 

0.1216  0.0618 

0.0150 

0.0032 

-0.0095 

-0.0326 

B 

0.0618-  0.0410 

0.0038 

-0.0066 

-0.0169 

-0.0348 

Z( 

1) 

0.0150  0.0038 

0.0159 

0.0128 

0.0112 

0.0092 

Z( 

2) 

0.0032  -0.0066 

0.0128 

0.0233 

0.0235 

0.0254 

Z( 

3) 

-0.0095  -0.0169 

0.0112 

0.0235 

0.0350 

0.0406 

Z( 

4) 

-0.0326  -0.0348 

0.0092 

0.0254 

0.0406 

0.0667 

CORRELATION  MATRIX: 

A 

1.0000  0.8756 

0.3411 

0.0600 

-0.1452 

-0.3622 

B 

0.8756  1.0000 

0.1502 

-0.2124 

-0;4452 

-0.6656 

Z( 

1) 

0.3411  0.1502 

1.0000 

0.6645 

0.4752 

0.2833 

Z( 

2) 

0.0600  -0.2124 

0.6645 

1.0000 

0.8240 

0.6448 

Z( 

3) 

-0.1452  -0.4452 

0.4752 

0.8240 

1.0000 

0.8409 

Z( 

4) 

-0.3622  -0.6656 

0.2833 

0.6448 

0.8409 

1.0000 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF  TPF 


(LOWER  BOUND,  UPPER  BOUND) 


0.005 

0.5118 

0.010 

(j.5803 

0.020 

0.6524 

0.030 

0.6956 

0.040 

0.7264 

0.050 

0.7503 

0.060 

0.7697 

0.070 

0.7861 

0.080 

0.8001 

0.090 

0.8123 

0.100 

0.8232 

0.110 

0.8329 

0.120 

0.8417 

0.130 

0.8498 

0.140 

0.8571 

0.150 

0.8639 

0.200 

0.8912 

0.250 

0.9114 

0.300 

0.9269 

(  0.3057 
(  0.3971 
(  0.4987 
(  0.5595 
(  0.6018 
(  0.6334 
(  0.6581 
(  0.6781 
(  0.6947 
(  0.7088 
(  0.7209 
(  0.7315 
(  0.7409 
(  0.7493 
(  0.7569 
(  0.7638 
(  0.7911 
(  0.8111 
(  0.8269 


0.7147  ) 
0.7473  ) 
0.7843  ) 
0.8089  ) 
0.8279  ) 
0.8438  ) 
0.8574  ) 
0.8693  ) 
0.8799  ) 
0.8894  ) 
0.8979  ) 
0.9056  ) 
0.9126  ) 
0.9190  ) 
0.9248  ) 
0.9302  ) 
0.9511  ) 
0.9654  ) 
0.9753  ) 


0.400  0.9496 

(  0.8517 

/ 

0.9874  ) 

.  .  0.500  0.9654 

(  0.8715 

r 

0.9938  ) 

0.600  0.9768 

(  0.8887 

9 

0.9972  ) 

0.700  0.9854 

(  0.9047 

f 

0.9989  ) 

0.800  0.9918 

(  0.9210 

9 

0.9997  ) 

0.900  0.9966 

(  0.9395 

9 

0.9999  ) 

0.950  0.9985 

(  0.9519 

9 

1.0000  ) 

ESTIMATES  OF  EXPECTED 

OPERATING  POINTS 

ON  FITTED  ROC 

CURVE,  WITH  LOWER  AND 

UPPER  BOUNDS 

OF  ASYMMETRIC  95% 

CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS 

• 

EXPECTED  OPERATING  POINT 

LOWER  BOUND 

UPPER  ] 

BOUND 

(  FPF  ,  TPF  )  • 

(  FPF  , 

TPF 

)  (  FPF  , 

TPF  ) 

(0.0124,  0.6021) 

(0.0030,  0 

.4632)  (0.0410, 

0.7291) 

(0.0317,  0.7017) 

(0.0131,  0 

.6083)  (0.0682, 

0.7834) 

(0.0602,  0.7702) 

(0.0320,  0 

.7026)  (0.1049, 

0.8282) 

(0.1463,  0.8614) 

(0.0968,  0 

.8199)  (0.2103, 

0.8958) 

1 

R  0  C  F  I  T 

(JUNE 

1993  VERSION) 

: 

MAXIMUM  LIKE 

L  I  H  0  0  D 

E  S 

T  I  M  A  T  I  0 

N 

OF  A  B  I  N  0 

R  M  A  L  R  ( 

3  C 

CURVE 

FROM  RATING  DATA 


DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMA: 


NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149. 


NO.  OF  ACTUALLY  POSITIVE  CASES  =  27. 


RESPONSE  DATA: 


CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

1 

93. 

6. 

2 

27, 

0. 

3 

10. 

2. 

4 

10. 

8. 

5 

9. 

11. 

OBSERVED  OPERATING  POINTS: 
FPF:  0.0000  0.0604  0.1275 
TPF:  0.0000  0.4074  0.7037 

0.1946 

0.7778 

0.3758 

0.7778 

1.0000 

1.0000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.3079  B=  0.8539 

Z(K)=  0.3160  0.8608  1.1383  1.5517 

LOGL=  -209.1670 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AF 


PROCEDURE  CONVERGES  AFTER  5  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.1742  B=  0.8234 

Z{K)=  0.3269  0.8058  1.0795  1.6264 

LOGL=  -208.0945 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AR 


A 


VARIANCE-COVARIANCE  MATRIX: 

0.1161  0.0609  0.0099  0.0079  0.0057  -0.0022 


B 

0.0609 

0.0596 

0.0035 

-0.0003 

-0.0037 

-0.0152 

Z( 

1) 

0.0099 

0.0035 

0.0109 

0.0082 

0.0071 

0.0053 

Z( 

2) 

0.0079 

-0.0003 

0.0082 

0.0129 

0.0114 

0.0095 

Z( 

3) 

0.0057 

-0.0037 

0.0071 

0.0114 

0.0155 

0.0134 

Z( 

4) 

-0.0022 

-0.0152 

0.0053 

0.0095 

0.0134 

0.0283 

A 

CORRELATION  MATRIX: 
1.0000  0.7313  0.2783 

0.2036 

0.1354 

-0.0388 

B 

0.7313 

1.0000 

0.1356 

-0.0097 

-0.1210 

-0.3698 

Z( 

1) 

0.2783 

0.1356 

1.0000 

0.6919 

0.5454 

0.3029 

Z( 

2) 

0.2036 

-0.0097 

0.6919 

1.0000 

0.8055 

0.4983 

Z( 

3) 

0.1354 

-0.1210 

0.5454 

0.8055 

1.0000 

0.6430 

Z( 

4) 

-0.0388 

-0.3698 

0.3029 

0.4983 

0.6430 

1.0000 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASTI^TRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.1718 

( 

0.0344  , 

0.4705 

) 

0.010 

0.2291 

( 

0.0648  , 

0.5128 

) 

0.020 

0.3025 

( 

0.1171  , 

0.5617 

). 

0.030 

0.3539 

( 

0.1613  , 

0.5946 

) 

0.040 

0.3945 

( 

0.1999  , 

0.6205 

) 

0.050 

0.4284 

( 

0.2341  , 

0.6422 

) 

0.060 

0.4577 

( 

0.2647  , 

0.6614 

) 

0.070 

0.4836 

( 

0.2924  , 

0.6786 

) 

0.080 

0.5068 

( 

0.3176  , 

0.6944 

) 

0.090 

0.5279 

( 

0.3407  , 

0.7091 

) 

0.100 

0.5473 

( 

0.3618  , 

0.7228 

) 

0.110 

0.5652 

.  ( 

0.3813  , 

0.7357 

) 

0.120 

0.5819 

( 

0.3994  , 

0.7480 

) 

0.130 

0.5974 

( 

0.4162  , 

0.7596 

) 

0.140 

0.6120 

( 

0.4318  , 

0.7707 

) 

0.150 

0.6258 

( 

0.4464  , 

0.7812 

) 

0.200 

0.6849 

( 

0.5071  , 

0.8276 

) 

0.250 

0.7321 

( 

0.5534  , 

0.8652 

) 

0.300 

0.7712 

( 

0.5905  , 

0.8955 

) 

0.400 

0.8330 

( 

0.6483  , 

0.9396 

) 

O'.  500 

0.8798 

( 

0.6937  , 

0.9673 

) 

0.600 

0.9166 

( 

0.7329  , 

0.9840 

) 

0.700 

0.9458 

( 

0.7695  , 

0.9933 

) 

0.800 

0.9691 

( 

0.8068  , 

0.9979 

) 

0.900 

0.9871 

( 

0.8502  , 

0.9997 

) 

0.950 

0.9943 

( 

0.8798  , 

0.9999 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 
{  FPF  ,  TPF  ) 


LOWER  BOUND  UPPER  BOUND 

(  FPF  ,  TPF  )  (  FPF  ,  TPF  ) 

(0.0252,  0.3313) 


(0.0519,  0.4345) 


(0.0974,  0.5424) 


(0.1402,  0.6123) 
(O'. 2102,  0.6952) 
(0.3719,  0.8173) 


(0.0929,  0.5337)  (0.2016,  0.6865) 

(0.1519,  0.6284)  (0.2798,  0.7561) 

(0.2974,  0.7692)  (0.4515,  0.8586) 

R  0  C  F  I  T  (JUNE  1993  VERSION)  : 

MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  COLLECTED  IN  5  CJVTEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMA 


NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  150. 


RESPONSE  DATA: 

CATEGORY  1 
ACTUALLY  NEGATIVE  CASES  28. 
ACTUALLY  POSITIVE  CASES  3. 


2 

62. 

23. 


3  4  ,5 

59.  0.  0. 

78.  33.  13. 


OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0000  0.0000  0.3960  0.8121  1.0000 
TPF:  0.0000  0.0867  0.3067  0.8267  0.9800  1.0000 


INITIAL  VALUES  OF  PARAMETERS: 

A=  1.2603  B=  0.8325 

Z(K)=  -0.8855  0.2634  2.6112  2.7112 

LOGL=  -397.9249 

CHI-SQUARE  GOODNESS  OF  FIT  NOT, CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  A 


PROCEDURE  CONVERGES  AFTER  7  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.2232  B=  0.6858 

Z(K)-  -0.9217  0.3090  2.5500  3.7878 

LOGL=  -345.4767 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  A: 


VARIANCE-COVARIANCE  MATRIX: 


A 

0.0189 

0.0045 

0.0052 

0.0060 

0.0056 

-0.0005 

B 

0.0045 

0.0099 

0.0027 

-0.0001 

-0.0269 

-0.0459 

Z( 

1) 

0.0052 

0.0027 

0.0140 

0.0051 

-0.0023 

-0.0073 

2( 

2) 

0.0060 

-0.0001 

0.0051 

0.0103 

0.0087 

0.0091 

Z( 

3) 

0.0056 

-0.0269 

-0.0023 

0.0087 

0.1115 

0.1582 

Z( 

4) 

-0.0005 

-0.0459 

-0.0073 

0.0091 

0.1582 

0.2801 

CORRELATION  MATRIX: 


A 

1.0000 

0.3270 

0.3178 

0.4283 

0.1216 

-0.0072 

B 

0.3270 

1.0000 

0.2319 

-0.0120 

-0.8074 

-0.8706 

Z( 

1) 

0.3178 

0.2319 

1.0000 

0.4245 

-0.0585 

-0.1170 

Z( 

2) 

0.4283 

-0.0120 

0.4245 

1.0000 

0.2583 

0.1695 

Z( 

3) 

0.1216 

-0.8074 

-0.0585 

0.2583 

1.0000 

0.8953 

Z( 

4) 

-0.0072 

-0.8706 

-0.1170 

0.1695 

0.8953 

1.0000 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  AS™METRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 


FALSE-POSITIVE  FRACTION: 


FPF  TPF 


(LOWER  BOUND,  UPPER  BOUND) 


0.005 

0.2934 

0.010 

0.3548 

0.020 

0.4264 

0.030 

0.4734 

0.040 

0.5089 

0.050 

0.5379 

0.060 

0.5623 

0.070 

0.5836 

0.080 

0.6024 

0.090 

0.6193 

0.100 

0.6347 

0.110 

0.6488 

0.120 

0.6618 

0.130  . 

0.6739 

0.140 

0.6852 

0.150 

0.6958 

0.200 

0.7409 

0.250 

0.7766 

0.300 

0.8062 

0.400 

0.8531 

0.500 

0.8894 

0.600 

0.9187 

0.700 

0.9432 

0.800 

0.9641 

0.900 

0.9822 

0.950 

0.9906 

( 

( 

( 

( 

( 

( 

( 

( 

( 

( 

( 

( 

( 

( 

( 

( 

( 

( 

( 

( 

( 

( 

( 

( 

( 

( 


0.1514  , 

0.2065  , 

0.2779  , 

0.3282  , 

0.3680  , 

0.4012  , 

0.4298  , 

0.4550  , 

0.4776  , 

0.4981  , 

0.5169  , 

0.5341  , 

0.5502  , 

0.5651  , 

0.5791  .  , 
0.5922  , 

0.6482  , 

0.6924  , 

0.7287  , 

0.7859  , 

0.8299  , 

0.8659  , 

0.8970  , 

0.9253  , 

0.9534  , 

0.9692  , 


0.4775  ) 
0.5294  ) 
0.5864  ) 
0.6222  ) 
0.6488  ) 
0.6702  ) 
0.6881  ) 
0.7037  ) 
0.7174  ) 
0.7297  ) 
0.7409  ) 
0.7512  ) 
0.7608  ) 
0.7696  ) 
0.7779  ) 
0.7857  ) 
0.8191  ) 
0.8460  ) 
0.8684  ) 
0.9045  ) 
0.9322  ) 
0.9541  ) 
0.9713  ) 
0.9846  ). 
0.9942  ) 
0.9977  ) 


ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT  LOWER  BOUND  UPPER  BOUND 


(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(0.0001, 

(0.0054, 

(0.3787, 

(0.8217, 

0.0847) 

0.2996) 

0.8441) 

0.9682) 

(0.0000, 

(0.0007, 

(0.3059, 

(0.7548, 

0.0185) 

0.1650) 

0.8093) 

0.9551) 

(0.0030, 

(0.0290, 

(0.4560, 

(0.8757, 

0.2537) 

0.4695) 

0.8744) 

0.9780) 

R  0  C  F  I  T  (JUNE  1993  VERSION)  : 

MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTIONjU^milimHlllllimPi 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMAL 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  59. 

RESPONSE  DATA: 

CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000.0.0067  0.0470  0.0537  0.3289  1.0000 
TPF:  0.0000  0.5763  0.7458  0.7966  0.8983  1.0000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.5514  B=  0.5243 

Z(K)=  0.4426  1.6104  1.6752  2.4728 

LOGL=  -197.8084 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  C:ALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AR 


PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.5534  B=  0,5279 

Z(K)=  0.4474  1.5513  1.7104  2.5634 

LOGL=  -196.4182 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AR 


VARIANCE-COVARIANCE  MATRIX: 

A  0.0741  0.0289  0.0066  0.0035  0.0017  -0.0209 

B  0.0289  0.0197  0.0019  -0.0042  -0.0064  -0.0298 

Z(  1)  0.0066  0.0019  0.0113  0.0065  0.0060  0.0035 

Z(  2)  0.0035  -0.0042  0.0065  0.0246  0.0240  0.0263 

Z(  3)  0.0017  -0.0064  0.0060  0.0240  0.0296  0.0336 

Z(  4)  -0.0209  -0.0298  0.0035  0.0263  0.0336  0.1096 


CORRELATION  MATRIX: 

A  1.0000  0.7573  0.2287  0.0831  0.0367  -0.2316 

B  0.7573  1.0000  0.1262  -0.1906  -0.2655  -0.6429 

Z(  1)  0.2287  0.1262  1.0000  0.3888  0.3294  0.0989 

Z(  2)  0.0831  -0.1906  0.3888  1.0000  0.8887  0.5059 

Z(  3)  0.0367  -0.2655  0.3294  0.8887  1.0000  0.5900 

Z(  4)  -0.2316  -0.6429  0.0989  0.5059  0.5900  1.0000 


ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


1  2  3  4  5 

100.  41.  1.  6.  1, 

6.  6.  3.  10.  34. 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.5767 

( 

0.3939 

f 

0.7440 

) 

0.010 

0.6274 

( 

0.4620 

9 

0.7721 

) 

0.020 

0.6805 

( 

0.5340 

9 

0.8031 

) 

0.030 

0.7124 

( 

0.5769 

9 

0.8229 

) 

0.040 

0.7353 

( 

0.6072 

9 

0.8379 

) 

0.050 

0.7533 

( 

0.6305 

9 

0.8500 

) 

0.060 

0.7681 

( 

0.6492 

9 

0.8603 

) 

0.070 

0.7806 

( 

0.6649 

9 

0.8692 

) 

0.080 

0.7915 

( 

0.6782 

9 

0.8770 

) 

0.090 

0.8011 

( 

0.6898 

9 

0.8841 

) 

0.100 

0.8097 

( 

0.7001 

9 

0.8905 

) 

0.110 

0.8175 

( 

0.7092 

9 

0.8963 

) 

0.120 

0.8246 

( 

0.7175 

9 

0.9016 

) 

0.130 

0.8312 

( 

0.7250 

9 

0.9065 

) 

0.140 

0.8372 

( 

0.7319 

9 

0.9111 

) 

0.150  . 

0.-8429 

( 

0.7383 

9 

0.9153 

) 

0.200 

O; 8663 

( 

0.7643- 

9 

0.9329 

) 

0.250 

0.8844 

( 

0.7841 

9 

0.9462 

) 

0.300 

0.8992 

( 

0.8001 

9 

0,9565 

) 

0.400 

0.9222 

( 

0.8256 

9 

0.9715 

) 

0.500 

0.9398 

( 

0.8461 

9 

0.9816 

) 

0.600 

0.9542 

( 

0.8641 

9 

0.9885 

) 

0.700 

0.9664 

( 

0.8811 

9 

0.9934 

) 

0.800 

0.9771 

( 

0.8985 

9 

0.9968 

) 

0.900 

0.9871 

( 

0.9188 

9 

0.9989 

) 

0.950 

0.9923 

( 

0.9329 

9 

0.9996 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT  LOWER  BOUND  UPPER  BOUND 


(  FPF  , 

TPF  ) 

(  FPF  ,  TPF  ) 

(  FPF  , 

TPF  ) 

(0.0052, 

0.5793) 

(0.0007,  0.4434) 

(0.0278, 

0.7063) 

(0.0436, 

0.7423) 

(0.0203,  0.6817) 

(0.0848, 

0.7963) 

(0.0604, 

0.7687) 

(0.0315,  0.7164) 

(0.1068, 

0.8151) 

(0.3273, 

0.9061) 

(0.2559,  0.8863) 

(0.4056, 

0.9233) 

R 

0  C  F  I  T  (JUNE  1993  VERSION) 

■ 

MAXI 

MUM 

L 

I 

K  E 

L 

IHOOD  EST 

I  M  A  T  I  0 

N 

0 

F  A 

B 

I 

N  0 

R 

M  A  L  ROC  C 

U  R  V  E 

F  H 

0 

M 

R 

A 

TING  DATA 

DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMAL 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  82. 

RESPONSE  DATA: 

CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0268  0.0268  0.0268  0.0940  1.0000 


1 

135. 

29. 


2 

10. 

1. 


3 

0. 

1. 


4 

0. 

5. 


5 

4. 

46. 


TPF:  0.0000  0.5610  0.6220  0.6341  0.6463  1.0000 


INITIAL  VALUES  OF  PARAMETERS: 

A-  0.7642  B=  0.2756 

Z(K)=  1.3170  1.7297  1.8297  1.9297 

LOGL=  -142.8031 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  K 


PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  0.7916  B=  0.3089 

Z(K)=  1,3225  1.7410  1.7930  2.0498 

LOGL=  -140.5911 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AI 


A 

VARIANCE-COVARIANCE  MATRIX: 
0.0733  0.0328  0.0093  -0.0032 

-0.0052 

-0.0172 

B 

0.0328 

0.0204 

0.0023 

-0.0066 

-0.0080 

-0,0164 

Z( 

1) 

0.0093 

0.0023 

0.0204 

0.0171 

0.0168 

0.0150 

Z( 

2) 

-0.0032 

-0.0066 

0.0171 

0.0314 

0.0315 

0,0327 

Z( 

3) 

-0.0052 

-0,0080 

0.0168 

0.0315 

0.0340 

0.0357 

Z( 

4) 

-0.0172 

-0.0164 

0.0150 

0.0327 

0.0357 

0.0536 

CORRELATION  MATRIX: 


A 

1.0000 

0.8490 

0.2413 

-0.0675 

-0.1051 

-0.2742 

B 

0.8490 

1.0000 

0.1123 

-0.2591 

-0.3030 

-0.4970 

Z( 

1) 

0.2413 

0.1123 

1.0000 

0.6764 

0.6365 

0.4546 

Z( 

2) 

-0.0675 

-0.2591 

0.6764 

1.0000 

0.9640 

0.7976 

Z( 

3) 

-0.1051 

-0.3030 

0.6365 

0.9640 

1.0000 

0.8360 

Z( 

4) 

-0.2742 

-0.4970 

0.4546 

0.7976 

0.8360 

1.0000 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


-FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


0.005 

0.4983 

( 

0.010 

0.5290 

( 

0.020 

0.5624 

( 

0.030 

0.5833 

( 

0.040 

0.5989 

( 

0.050 

0.6115 

( 

0.060 

0.6222 

( 

0.070 

0.6314 

( 

0.080 

0.6396 

( 

0.090 

0.6470 

( 

0.100 

0.6538 

( 

0.110 

0.6601 

( 

0.120 

0.6659 

( 

0.130 

0.6713 

( 

0.140 

0.6765 

( 

0.150 

0.6813 

( 

0.200 

0.7025 

( 

0,250 

0.7202 

( 

0.300 

0.7356 

( 

0.3469 

r 

0.6499 

) 

0.3929 

r 

0.6618 

) 

0.4405 

r 

0.6785 

) 

0.4681 

r 

0.6918 

) 

0.4870 

9 

0.7032 

) 

0.5011 

9 

0.7136 

) 

0.5121 

9 

0.7231 

) 

0.5210 

9 

0.7319 

) 

0.5284 

9 

0.7401 

) 

0.5347 

9 

0.7478 

) 

0.5400 

9 

0.7551 

) 

0.5447 

9 

0.7620 

) 

0.5489 

9 

0.7686 

) 

0.5526 

9 

0.7749 

) 

0.5559 

9 

0.7808 

) 

0.5589 

9 

0.7865 

) 

0.5707 

9 

0.8119 

) 

0.5791 

9 

0.8332 

) 

0.5856 

9 

0.8516 

) 

0.400 

0.7622 

( 

0.500 

0.7857 

( 

0.600 

0.8078 

( 

0.700 

0.8298 

( 

0.800  - 

0.8535 

( 

0.900 

0.8825 

( 

0.950 

0.9032 

( 

0.5954  ,  0.8821  ) 
0.6029  ,  0.9070  ) 
0.6094  ,  0.9281  ) 
0.6155  ,  0.9466  ) 
0.6219  ,  0.9635  ) 
0.6298  ,  0.9795  ) 
0.6357  ,  0.9879  ) 


ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT  LOWER  BOUND  UPPER  BOUND 

(  FPF  ,  TPF  )  (  FPF  ,  .  TPF  )  (  FPF  ,  TPF  ) 

(0.0202,  0.5629)  (0.0062,  0.5073)  (0.0552,  0.6173) 

(0.0365,  0.5939)  (0.0156,  0.550,1)  (0.0761,  0.6366) 

(0.0408,  0.6001)  (0.0184,  0.5582)  (0.0817,  0.6410) 

(0.0930,  0.6491)  (0.0545,  0.6165)  (0.1487,  0.6807) 

1  R  0  C  F  I  T  (JUNE  1993  VERSION)  : 

MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMAL 
NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  27. 

RESPONSE  DATA: 

CATEGORY  1  2  3  4.5 

ACTUALLY  NEGATIVE  CASES  100.  14.  0.  21.  14. 

ACTUALLY  POSITIVE  CASES  11.  1.  0.  11.  4. 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0940  0.2349  0.3289  1.0000 
TPF:  0.0000  0.1481  0.5556  0.5926  1.0000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.0544  B=  1.5447 

2(K)=  0.4426  0.7225  1.3170 

LOGL=  -180.4581 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AR 


PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  0.9177  B=  1.3982 

Z(K)=  0.4487  0.6788  1.3417 

LOGL=  -179.8747 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AR 


VARIANCE-COVARIANCE  MATRIX: 

A  0.1578  0.1213  0.0189  0.0149  -0.0004 


B 

0.1213 

0.1589 

0.0070 

0.0008 

-0.0213 

Z( 

1) 

0.0189 

0.0070 

0.0113 

0.0099 

0.0069 

Z( 

2) 

0.0149 

0.0008 

0.0099 

0.0119 

0.0092 

Z( 

3) 

-0.0004 

-0.0213 

0.0069 

0.0092 

0.0207 

CORRELATION  MATRIX: 


A 

■  1.0000 

.0.7660 

0.4469 

0.3448 

-0.0065 

B 

0.7660 

1.0000 

0.1651 

0.0182 

-0.3724 

Z( 

1) 

0.4469 

0.1651 

1.0000 

0.8521 

0.4504 

Z( 

2) 

0.3448 

0.0182 

0.8521 

1.0000 

0.5831 

Z( 

3) 

-0.0065 

-0.3724 

0.4504 

0.5831 

1.0000 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


0.005 

0.0036 

( 

0.0000 

9 

0.1186 

) 

0.010 

0.0098 

( 

0.0001 

9 

0.1550 

) 

0.020 

0.0253 

( 

0.0010 

9 

0.2037 

) 

0.030 

0.0434 

( 

0.0033 

9 

0.2402 

) 

0.040 

0.0629 

( 

0.0071 

9 

0.2707 

) 

0.050 

0.0834 

( 

0.0127 

9 

0;2978 

) 

0.060 

0.1044 

( 

0.0201 

9 

0.3225 

) 

0.070 

0.1259 

( 

0.0290 

9 

0.3456 

) 

0.080 

0.1475 

( 

0.0395 

9 

0.3675 

) 

0.090 

0.1692 

( 

0.0514 

9 

0.3885 

) 

0.100 

0.1909 

( 

0.0645 

9 

0.4089 

) 

0.110 

0.2126 

( 

0.0785 

9 

0.4287 

) 

0.120 

0.2341 

( 

0.0933 

9 

0.4482 

) 

0.130 

0.2555 

'  ( 

0.1088 

9 

0.4673 

) 

0.140 

0.2766 

( 

0.1248 

9 

0.4863 

) 

0.150 

0.2975 

( 

0.1410 

9 

0.5051 

) 

0.200 

0.3979 

( 

0.2227 

9 

0.5969 

) 

0.250 

0.4900 

( 

0.2980 

9 

0.6845 

) 

0.300 

0.5734 

( 

0.3635 

9 

0.7640 

) 

0.400 

0.7136 

( 

0.4698 

9 

0.8857 

) 

0.500 

0.8206 

( 

0.5554 

9 

0.9551 

) 

0.600 

0.8982 

( 

0.6304 

9 

0.9864 

) 

0.700 

0.9506 

{ 

0.7009 

9 

0.9972 

) 

0.800 

0.9819 

( 

0.7716 

9 

0.9997 

) 

0.900 

0.9966 

( 

0.8498 

9 

1.0000 

) 

0.950 

0.9994 

( 

0.8981 

9 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 
(  FPF  ,  TPF  ) 

LOWER  BOUND 
(  FPF  ,  TPF  ) 

UPPER  BOUND 
(  FPF  ,  TPF  ) 

(0.0898, 

(0.2486, 

(0.3268, 

0.1689) 

0.4875) 

0.6142) 

(0.0522,  0.0881) 
(0.1860,  0.3705) 
(0.2554,  0.4994) 

(0.1446,  0.2863) 
(0.3210,  0.6056) 
(0.4051,  0.7197) 

R  0  C  F  I  T  (JUNE  1993  VERSION)  : 

MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
•  FROM  RATING  DATA 


DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMAI 
NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  150. 

RESPONSE  DATA: 

CATEGORY  12 

ACTUALLY  NEGATIVE  CASES  93.  42. 

ACTUALLY  POSITIVE  CJ^SES  24.  42. 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000-0.0000  0.0336  0.0940  0.3758  1.0000 
TPF:  0.0000  0.1333  0.3133  0.5600  0.8400  1.0000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.2702  B=  0.8959 

Z(K)=  0.3160  1.3170  1.8313  2.7112 

LOGL=  -378.3870 

^^^~SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AF 
PROCEDURE  CONVERGES  AFTER  5  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.2787  B=  0.8858 

Z(K)=  0.3172  1.2914  1.9619  2.7109 

LOGL=  -376.3607 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AR 
VARIANCE -COVARIANCE  MATRIX: 


A 

0.0312 

0.0155 

0.0106 

0.0052 

-0.0025 

-0.0137 

B 

0.0155 

0.0171 

0.0034 

-0.0060 

-0.0167 

-0.0309 

Z( 

1) 

0.0106 

0.0034 

0.0109 

0.0067 

0.0044 

0.0016 

Z( 

2) 

0.0052 

-0.0060 

0.0067 

0.0165 

0.0189 

0.0234 

Z( 

3) 

-0.0025 

-0.0167 

0.0044 

0.0189 

0.0361 

0.0480 

Z( 

4) 

-0.0137 

-0.0309 

0.0016 

0.0234 

0.0480 

0.0866 

CORRELATION  MATRIX: 

A 

1.0000 

0.6727 

0.5739 

0.2270 

-0.0744 

-0.2638 

B 

0.6727 

1.0000 

0.2506 

-0.3583 

-0.6697 

-0.8032 

Z( 

1) 

0.5739 

0.2506 

1.0000 

0.4988 

0.2195 

0.0519 

Z( 

2) 

0.2270 

-0.3583 

0.4988 

1.0000 

0.7734 

0.6186 

Z( 

3) 

-0.0744 

-0.6697 

0.2195 

0.7734 

1.0000 

0.8585 

Z( 

4) 

-0.2638 

-0.8032 

0.0519 

0.6186 

0.8585 

1.0000 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


3  4  5 

9.  5.  0. 

37.  27.  20. 


FPF  TPF 


(LOWER  BOUND,  UPPER  BOUND) 


0.005 

0.1578 

0.010 

0.2170 

0.020 

0.2943 

0.030 

0.3491 

0.040 

0.3926 

0.050 

0.4291 

0.060 

0.4606 

0.070 

0.4885 

0.080 

0.5135 

0.090 

0.5362 

0.100 

0.5570 

0.110 

0.5761 

0.120 

0.5940 

0.130 

0.6106 

0.140 

0.6261 

0.150 

0.6408 

0.200 

0..7031 

0.250 

0.7522 

0.300 

0.7923 

0.400 

0.8542 

0.500 

0.8995 

0.600 

0.9335 

0.700 

0.9593 

0.800 

0.9785 

0.900 

0.9921 

0.950 

0.9969 

(  0.0665 
(  0.1098 
(  0.1759 
(  0.2280 
(  0.2717 
(  0.3095 
(  0.3431 
(  0.3731 
(  0.4004 
(  0.4253 
(  0.4482 
(  0.4694 
(  0.4891 
(  0.5075 
(  0.5247 
(  0.5408 
(  0.6090 
(  0.6622 
{  0.7055 
(  0.7729 
(  0.8245 
(  0.8665 
(  0.9024 
(  0.9342 
(  0.9638 
(  0.9790 


0.3070  ) 
0.3680  ) 
0.4400  ) 
0.4880  ) 
0.5251  ) 
0.5556  ) 
0.5818  ) 
0.6048  ) 
0.6255  ) 
0.6443  ) 
0.6616  ) 
0.6775  ). 
0.6924  ) 
0.7064  ) 
0.7195  ) 
0.7319  ) 
0.7852  ) 
0.8275  ) 
0.8619  ) 
0.9132  ) 
0.9479  ) 
0.9710  ) 
0.9858  ) 
0.9945  ) 
0.9988  ) 
0.9997  ) 


ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT  LOWER  BOUND  UPPER  BOUND 


(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(0.0034, 

(0.0249, 

(0.0983, 

(0.3755, 

0.1308) 

0.3230) 

0.5536) 

0.8408) 

(0.0005, 

(0.0098, 

(0.0613, 

(0.3010, 

0.0512) 

0.2150) 

0.4647) 

0.7929) 

(0.0164, 

(0.0560, 

(0.1493, 

(0.4551, 

0.2704) 

0.4486) 

0.6398) 

0.8807) 

R  0  C  F  I  T  (JUNE  1993  VERSION)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION 


DATA  COLLECTED  IN  5  <y^TEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 


(E.G., 


THAT  ABNORMAL 


NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  59 

RESPONSE  DATA: 

CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0134  0.0671  0.3356  0.7450  1.0000 
TPF:  0.0000  0.5424  0.6441  0.9153  0.9661  1.0000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.4749  B=  0.6394 

Z(K)=  -0.6584  0.4241  1.4979  2.2142 

LOGL=  -263.4115 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ART 


1 

38. 

2. 


2 

61. 

3. 


3 

40. 

16. 


4 

8. 

6. 


5 

2. 

32. 


PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.4939  B=  0.6633 

Z(K)=  -0.6526  0.4028  1.5744  2.1027 

LOGL=  -262.5013 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARI 


A 

B 

Z(  1) 
Z(  2) 
Z(  3) 
?(  4) 


VARIANCE-COVARIANCE  MATRIX: 


0.0536  0.0202 

0.0202  0.0175 

0.0051  0.0020 

0.0058  0.0010 

0.0046  -0.0061 
-0.0010  -0.0151 


0.0058  0.0046  -0.0010 

0.0010  -0.0061  -0.0151 


0.0051 

0.0020 

0.0122 

0.0052 

0.0026 

0.0015 


0.0052 

0.0108 

0.0064 

0.0055 


0.0026 

0.0064 

0.0251 

0.0257 


0.0015 

0.0055 

0.0257 

0.0492 


A 

B 

Z(  1) 
Z(  2) 
Z(  3) 
Z(  4) 


CORRELATION  MATRIX: 


1.0000  0.6613 
0.6613  1.0000 

0.1977  0.1374 

0.2416  0.0691 

0.1252  -0.2914 
-0.0192  -0.5166 


0.1977 

0.1374 

1.0000 

0.4489 

0.1491 

0.0602 


0.2416 

0.0691 

0.4489 

1.0000 

0.3873 

0.2376 


0.1252 

-0.2914 

0.1491 

0.3873 

1.0000 

0.7321 


-0.0192 

-0.5166 

0.0602 

0.2376 

0.7321 

1.0000 


ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 


TPF 


(LOWER  BOUND,  UPPER  BOUND) 


0.005 

0.4149 

( 

0.010 

0.4803 

( 

0.020 

0.5523 

( 

0.030 

0.5972 

( 

0.040 

0.6302 

( 

0.050 

0.6564 

( 

0.060 

0.6781 

( 

0.070 

0.6967 

( 

0.080 

0.7129 

( 

0.090 

0.7272 

( 

0.100 

0.7401 

( 

0.110 

0.7518 

( 

0,120 

0.7625 

( 

0.130 

0.7724 

( 

0,140 

0,7815 

( 

0.150 

0,7900 

( 

0.200 

0;8253 

( 

0.250 

0.8524 

( 

0.300 

0.8742 

( 

0.400 

0.9076 

( 

0.500 

0.9324 

( 

0.600 

0.9517 

( 

0.700 

0.9672 

( 

0.800 

0.9799 

( 

0.900 

0.9905 

( 

0.950 

0.9951 

( 

0.2371  ,  0.6125  ) 
0.3068  ,  0.6577  ) 
0.3895  ,  0.7066  ) 
0.4434  ,  0.7371  ) 
0.4837  ,  0.7598  ) 
0.5159  ,  0.7780  ) 
0.5426  ,  0.7933  ) 
0.5654  ,  0.8065  ) 
0.5853  ,  0.8181  ) 
0.6029  ,  0.8284  ) 
0.6187  ,  0.8378  ) 
0.6329  ,  0.8464  ) 
0.6459  ,  0.8542  ) 
0.6578  ,  0.8615  ) 
0.6688  ,  0.8683  ) 
0.6789  ,  0.8746  ) 
0.7210.  ,  0.9007  ) 
0.7531  ,  0.9206  ) 
0.7789  ,  0.9363  ) 
0.8194  ,  0.9590  ) 
0.8509  ,  0.9743  ) 
0.8774  ,  0.9847  ) 
0.9012  ,  0.9917  ) 
0.9240  ,  0.9962  ) 
0.9480  ,  0.9989  ) 
0.9627  ,  0.9996  ) 


ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT  LOWER  BOUND  UPPER  BOUND 


(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(0.0177, 

(0.0577, 

(0.3436, 

(0.7430, 

0.5395) 

0.6735) 

0,8900) 

0.9730) 

(0.0056, 

(0.0297, 

(0.2720, 

(0.6685, 

0.4250) 

0.5962) 

0.8624) 

0.9627) 

(0.0477, 

(0.1032, 

(0,4213, 

(0.8077, 

0.6509) 

0.7440) 

0.9134) 

0.9808) 

R  0  C  F  I  T  (JUNE  1993  VERSION)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  roc  CURVE 
FROM  RATING  DATA 


DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMAL 
NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  82. 

RESPONSE  DATA: 

CATEGORY  1 

ACTUALLY  NEGATIVE  CASES  34, 

ACTUALLY  POSITIVE  CASES  3. 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0201  0.0336  0.1745  0.7718  1.0000 


2  3  4  5 

89.  21.  2.  3. 

27.  12.  7.  33. 


TPF:  0.0000  0.4024  0.4878  0.6341  0.9634  1,0000 


INITIAL  VALUES  OF  PARAMETERS: 

A-  1.1938  B-  0.7160 

Z(K)-  -0.7446  0.9365  1.8313  2.0514 

LOGL-  -270.8926 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL 


FREQUENCIES  AR 


PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.1283  B=  0.7053 

Z(K)=  -0.7614  0.9868  1.6973  1.9788 

LOGL=  -269.4543 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AK 


A 

B 

Z(  1) 
Z(  2) 
Z(  3) 
Z(  4) 


VARIANCE-COVARIANCE  MATRIX: 


0.0060  0.0065  0.0040  0.0022 

0.0028  -0.0015  -0.0078  -0.0114 


0.0311  0.0109 
0.0109  0.0120 
0.0060  0.0028 
0.0065  -0.0015 
0.0040  -0.0078 
0.0022  -0.0114 


0.0129  0.0037 
0.0037  0.0140 
0.0019  0.0125 
0.0011  0.0125 


0.0019  0.0011 
0.0125  0.0125 
0.0273  0.0283 
0.0283  0.0381 


A 

B 

Z(  1) 
Z(  2) 
Z(  3) 
Z(  4) 


CORRELATION  MATRIX: 


1.0000  0.5610 
0.5610  1.0000 
0.3014  0.2234 
0.3090  -0.1165 
0.1366  -0.4327 
0.0635  -0.5335 


0.3014 

0.2234 

1.0000 

0.2754 

0.1009 

0.0480 


0.3090 

-0.1165 

0.2754 

1.0000 

0.6388 

0.5420 


0;1366 

-0.4327 

0.1009 

0.6388 

1.0000 

0.8771 


0.0635 

-0.5335 

0.0480 

0.5420 

0.8771 

1.0000 


ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


0.005  0.2455  ( 
0.010  0.3040  ( 
0.020  0.3743  ( 
0.030  0,4213  ( 
0.040  0.4575  ( 
0.050  0.4872  ( 
0.060  0.5125  ( 
0.070  0.5347  ( 
0.080  0.5545  ( 
0.090  0.5724  ( 
0.100  0.5887  ( 
0.110  0.6037  ( 
0.120  0.6177  ( 
0.130  0.6307  ( 
0.140  0.6429  ( 
0.150  0.6544  ( 
0.200  0.7036  ( 
0.250  0.7430  ■  ( 
0.300  0.7760  ( 


0.1253 

r 

0.4095 

) 

0.1756 

r 

0.4627 

) 

0.2422 

f 

0.5231 

) 

0.2898 

/ 

0.5623 

) 

0.3277 

f 

0.5920 

) 

0.3595 

/ 

0.6162 

) 

0.3869 

r 

0.6369 

) 

0.4112 

/ 

0.6550 

) 

0.4330 

9 

0.6711 

) 

0.4527 

t 

0.6857 

) 

0,4708 

9 

0.6991 

) 

0.4875 

9 

0.7114 

) 

0.5030 

9 

0.7229 

) 

0.5174 

9 

0.7336 

) 

0.5310 

9 

0.7437 

) 

0.5437 

9 

0.7532 

) 

0.5981 

9 

0.7942 

) 

0.6415 

9 

0.8271 

) 

0.6777 

9 

0.8545 

) 

0.400 

0.8289 

( 

0.7362 

f 

0.8976 

0.500 

0.8704 

( 

0.7831 

/ 

0.9297 

0.600 

0.9043 

( 

0.8233 

i 

0.9540 

0.700 

0.9329 

( 

0.8596 

9 

0.9724 

‘  0.800 

0.9574 

( 

0.8944 

9 

0.9859 

0.900 

0.9789 

( 

0.9309 

9 

0.9951 

0.950 

0.9890 

( 

0.9526 

9 

0.9982 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 


LOWER  BOUND 


UPPER  BOUND 


(  FPF  , 

TPF  ■)' 

(  FPF  , 

•  TPF  ) 

(  FPF  , 

TPF  ) 

(0.0239, 

0.3946) 

(0.0091, 

0.2956) 

(0.0552, 

0.5009) 

(0.0448, 

0.4725) 

(0.0216, 

0.3831) 

(0.0848, 

0.5633) 

(0.1619, 

0.6672) 

(0.1114, 

0.6059) 

(0.2252, 

0.7244) 

(0.7768, 

0.9521) 

(0.7049, 

0.9342) 

(0.8375, 

0.9658) 

R 

0  C  F  I 

T  (JUNE  1993  VERSION) 

: 

MAXI 

MUM 

L 

I 

K  E 

L 

I  H  0  0 

D  E  S  T  I 

M  A  T  I  0 

N 

0 

F  A 

B 

I 

N  0 

R 

M  A  L 

ROC  C  U 

R  V  E 

F  R 

0 

M 

R 

A 

TING 

DATA 

DATA  DESCRIPTION 


DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF 


POSITIVITY 


(E.G., 


THAT  ABNORMAL 


NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149. 


NO.  OF  ACTUALLY  POSITIVE  CASES  = 


RESPONSE  DATA: 

CATEGORY  1 
ACTUALLY  NEGATIVE  CASES  22. 
ACTUALLY  POSITIVE  CASES  3. 


2 

58. 

6. 


3  4 

19.  26. 

3.  0. 


OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.1611  0.3356  0.4631  0.8523  1.0000 
TPF:  0.0000  0.5556  0.5556  0.6667  0.8889  1.0000 


5 

24. 

15. 


27. 


INITIAL  VALUES  OF  PARAMETERS: 

A=  0.5715  B=  0.5571 

Z(K)=  -1.0466  0.0924  0.4241  0  9900 

LOGL=  -260.0469 

CHI~SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED 


CELL  FREQUENCIES  AR: 


PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  0.5870  B=  0.5037 

Z(K)=  -1.0604  0.1050  0.4573  0.9660 

LOGL=  -259.7316 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED 


CELL  FREQUENCIES  AR] 


A 


VARIANCE-COVARIANCE  MATRIX: 

0.0526  0.0065  0.0041  0.0035  0.0036  0.0040 


1 


B 

0.0065 

0.0229 

0.0031 

0.0000 

-0.0010 

-0.0032 

Z( 

1) 

0.0041 

0.0031 

0.0159 

0.0051 

0.0039 

0.0027 

Z( 

2) 

0.0035 

0.0000 

0.0051 

0.0104 

0.0082 

0.0063 

Z( 

3) 

0.0036 

-0.0010 

0.0039 

0.0082 

0.0111 

0.0086 

Z( 

4) 

0.0040 

-0.0032 

0.0027 

0.0063 

0.0086 

0.0148 

CORRELATION  MATRIX: 

A 

1.0000 

0.1880 

0.1413 

0.1485 

0.1479 

0.1443 

B 

0.1880 

1.0000 

0.1599 

0.0013 

-0.0638 

-0.1762 

Z( 

1) 

0.1413 

0.1599 

1.0000 

0.3950 

0.2940 

0.1776 

Z( 

2) 

0.1485 

0.0013 

0.3950 

1.0000 

0.7640 

0.5067 

Z( 

3) 

0.1479 

-0.0638 

0.2940 

0.7640 

1.0000 

0.6694 

Z( 

4) 

0.1443 

-0.1762 

0.1776 

0.5067 

0.6694 

1.0000 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


0.005 

0.2386 

( 

0.010 

0.2793 

( 

0.020 

0.3272 

( 

0.030 

0.3592 

( 

0.040 

0.3840 

( 

0.050 

0.4045 

( 

0.060 

0.4222 

( 

0.070 

0.4378 

( 

0.080 

0.4519 

( 

0.090 

0.4647 

( 

0.100 

0.4766 

-  ( 

0.110 

0.4877 

( 

0.120 

0.4980 

( 

0.130 

0.5078 

( 

0.140 

0.5171 

( 

0.150 

0.5259 

( 

0.200 

0.5648 

( 

0.250 

0.5977 

( 

0.300 

0.6267 

( 

O'.  400 

0.6771 

( 

0.500 

0.7214 

( 

0.600 

0.7625 

,  ( 

0.700 

0.8026 

( 

0.800 

0.8439 

( 

0.900 

0.8911 

( 

0.950 

0.9216 

( 

0.0642 

r 

0.5396 

) 

0.0911 

f 

0.5652 

) 

0.1285 

9 

0.5940 

) 

0.1568 

9 

0.6128 

) 

0.1802 

9 

0.6272 

) 

0.2006 

9 

0.6391 

) 

0.2187 

9 

0.6495 

) 

0.2352 

9 

0.6586 

) 

0.2504 

9 

0.6670 

) 

0.2645 

9 

0.6746 

) 

0.2776 

9 

0.6817 

) 

0.2901 

9 

0.6884 

) 

0.3018 

9 

0.6947 

) 

0.3130 

9 

0.7008 

) 

0.3236 

9 

0.7065 

) 

0.3338 

9 

0.7120 

) 

0.3790 

9 

0.7371 

) 

0.4173 

9 

0.7592 

) 

0.4506 

9 

0.7794 

) 

0.5072 

9 

0.8162 

) 

0.5547 

9 

0.8500 

) 

0.5968 

9 

0.8817 

) 

0.6363 

9 

0.9120 

) 

0.6763 

9 

0.9411 

) 

0.7230 

9 

0.9695 

) 

0.7558 

9 

0.9838 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 
(  FPF  ,  TPF  ) 


LOWER  BOUND  UPPER  BOUND 

FPF  ,  TPF  )  (  FPF  ,  TPF  ) 


(0.1670,  0.5400) 


(0.1143,  0.4922) 


(0.2333,  0.5872) 


(0.3237, 

0.6393) 

(0.2533, 

0.5996) 

(0.4011, 

0.6775) 

(b.4582. 

0.7034) 

(0.3804, 

0.6677) 

(0.5377, 

0.7371) 

(0 . 8555, 

0.8689) 

(0.7919, 

0.8405) 

(0.9045, 

0.8936) 

R  0  C  F  I  T  (JUNE  1993  VERSION)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTIO 


DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G., 


THAT  ABNORMAL 


NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  150. 


RESPONSE. DATA: 

CATEGORY  1 
ACTUALLY  NEGATIVE  CASES  27. 
ACTUALLY  POSITIVE  CASES  7. 


2 

90. 

45. 


3  4 

32.  0. 

61.  25. 


OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0000  0.0000  0.2148  0.8188  1.0000 
TPF:  0.0000  0.0800  0.2467  0.6533  0.9533  1.0000 


5 

0. 

12. 


INITIAL  VALUES  OF  PARAMETERS: 

A=  0.9927  B=  0.7668 

Z(K)=  -0.9107  0.7898  2.6112  2.7112 

LOGL=  -384.2195 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AR 


PROCEDURE  CONVERGES  AFTER  7  ITERATIONS. 


FINAL  VALUES  OF. PARAMETERS : 

A=  0.9878  B=  0.6626 

Z(K)=  -0.9335  0.8314  2.5567  3.6335 

LOGL=  -347 . 6042 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE 


A 

B 

Z(  1) 
Z(  2) 
Z(  3) 
Z(  4) 


A 

B 

Z(  1) 
Z(  2) 
Z(  3) 
Z(  4) 


VARIANCE-COVARIANCE  MATRIX: 


0.0162  0.0042  0.0057 

0.0042  0.0074  0.0031 

0.0057  0.0031  0.0143 

0.0058  -0.0018  0.0038 

0.0026  -0.0188  -0.0029 
-0.0026  -0.0317  -0.0079 


CORRELATION  MATRIX: 
1.0000  0.3869  0.3724 

0.3869  1.0000  0.2980 

0.3724  0.2980  1.0000 

0.4085  -0.1812  0.2866 

0.0700  -0.7584  -0.0835 
-0.0449  -0.8210  -0.1477 


0.0058  0.0026  -0.0026 

-0.0018  -0.0188  -0.0317 
0.0038  -0.0029  -0.0079 
0.0126  0.0148  0.0178 

0.0148  0.0832  0.1112 

0.0178  0.1112  0.2013, 


0.4085  0.0700  -0.0449 

-0.1812  -0.7584  -0.8210 
0.2866  -0.0835  -0.1477 
1.0000  0.4581  0.3540 

0.4581  1.0000  0.8593 

0.3540  0.8593  1.0000 


ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 


FALSE-POSITIVE  FRACTION: 


TPF  (LOWER  BOUND,  UPPER  BOUND) 


0.005 

0.2360 

0.010 

0.2898 

0.020 

0.3545 

0.030 

0.3979 

0.040 

0.4315 

0,050 

0.4593 

0.060 

0.4830 

0.070 

0.5039 

0.080 

0.5226 

0,090 

0.5395 

0.100 

0.5551 

0.110 

0.5695 

0.120 

0,5829 

0.130 

0:5954 

0.140 

0,6072 

0.150 

0.6183 

0.200 

0.6665 

0.250 

0.7058 

0.300 

0.7391 

0.400 

0.7940 

0.500 

0.8384 

0.600 

0.8760 

0.700 

0.9091 

0,800 

0.9389 

0.900 

0.9669 

0.950 

0.9811 

( 

( 

( 

( 

{ 

( 

( 

( 

( 

( 

( 

( 

( 

( 

( 

( 

( 

( 

( 

( 

( 

( 

( 

( 

( 

( 


0.1295  , 

0,1764  , 

0.2378  , 

0.2817  , 

0.3167  , 

0.3463  , 

0.3720  , 

0.3949  , 

0.4156  , 

0.4344  , 

0.4518  , 

0.4680  , 

0.4830  , 

0.4971.  , 
0.5105  , 

0.5230  , 

0.5775  , 

0.6218  , 
0.6592  , 

0,7203  , 

0.7698  , 

0.8123  , 

0.8507  , 

0,8876  , 

0.9263  , 

0.9493  , 


0.3783  ) 
0.4291  ) 
0.4868  ) 
0.5241  ) 
0.5525  ) 
0.5756  ) 
0.5953  ) 
0,6126  ) 
0.6280  ) 
0.6419  ) 
0.6547  ) 
0.6666  ) 
0.6776  ) 
0.6879  ) 
0.697,7  ) 
0.7069  ) 
0.7470  ) 
0.7800  ) 
0.8082  ) 
0.8547  ) 
0.8921  ) 
0.9229  ) 
0.9485  ) 
0.9698  ) 
0.9870  ) 
0.9941  ) 


ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS; 


EXPECTED  OPERATING  POINT 
(  FPF  ,  TPF  ) 


LOWER  BOUND  UPPER  BOUND 

FPF  ,  TPF  )  (  FPF  ,  TPF  ) 


(0.0001,  0,0778) 
(0.0053,  0.2400) 
(0.2029,  0,6689) 
(0.8247,  0.9459) 


(0.0000,  0.0226) 
(0.0009,  0.1399) 
(0.1466,  0.6146) 
(0.7579,  0.9267) 


(0.0029,  0.2013) 
(0.0232,  0.3701) 
(0.2704,  0.7199) 
(0.8785,  0.9609) 


R  0  C  F  I  T  (JUNE  1993  VERSION)  : 

MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPMSENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMAL 
NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149.  NO.  OF  ACTUALLY  POSITIVE  CURSES  =  59. 

RESPONSE  DATA ; 

CATEGORY  12 

ACTUALLY  NEGATIVE  CASES  82.  51. 

ACTUALLY  POSITIVE  CASES  1.  3. 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0134  0.0537  0.1074  0.4497  1.0000 
TPF:  0.0000  0.6271  0.7797  0.8475  0.9831  1.0000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  2.1865  B=  0.8676 

Z(K)=  0.1262  1.2407  1.6104  2.2142 

LOGL=  -220.2996 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CIALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AF 


3  4  5 

8.  6.  2. 

4.  9.  37. 


PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  2.1233  B=  0.8333 

Z(K)=  0.1238  1.2597  1.6061  2.1666 

LOGL=  -220.1214 

CHI-SQUARE  GOODNESS  OF'  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AF 


VARIANCE-COVARIANCE  MATRIX: 


A 

0.1270 

0.0611 

0.0083 

0.0099 

0.0041 

-0.0168 

B 

0.0611 

0.0430 

0.0026 

-0.0018 

-0.0086 

-0.0286 

Z( 

1) 

0.0083 

0.0026 

0.0106 

0.0055 

0.0046 

0.0032 

Z( 

2) 

0.0099 

-0.0018 

0.0055 

0.0180 

0.0165 

0.0161 

Z( 

3) 

0.0041 

-0.0086 

0.0046 

0.0165 

0.0253 

0.0269 

Z( 

4) 

-0.0168 

-0.0286 

0.0032 

0.0161 

0.0269 

0.0560 

A 

CORRELATION  MATRIX: 
1.0000'  0.8274  0.2270 

0.2065 

0.0715 

-0.1987 

B 

0.8274 

1.0000 

0.1239 

-0.0647 

-0.2618 

-0.5828 

Z( 

1) 

0.2270 

0.1239 

1.0000 

0.4000 

0.2837 

0.1313 

Z( 

2) 

0.2065 

-0.0647 

0.4000 

1.0000 

0.7723 

0.5057 

Z( 

3) 

0.0715 

-0.2618 

0.2837 

0.7723 

1.0000 

0.7138 

Z( 

4) 

-0.1987 

-0.5828 

0.1313 

0.5057 

0.7138 

1.0000 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 


TPF 


(LOWER  BOUND,  UPPER  BOUND) 


0.005 

0.4907 

( 

0.010 

0.5732 

( 

0.020 

0.6597 

( 

0.030 

0.7108 

( 

0.040 

0.7467 

( 

0.050 

0.7741 

( 

0.060 

0.7960 

( 

0.070 

0.8142 

( 

0.080 

0.8295 

( 

0.090 

0.8428 

( 

0.100 

0.8543 

( 

0.110 

0.8646 

( 

0.120 

0.8737 

( 

0.130 

0.8819 

( 

0.140 

0.8893 

( 

0.150 

0.8961 

( 

0.200 

0.9225 

( 

0.250 

0.9408 

( 

0.300 

0.9542 

( 

0.400 

0.9721 

( 

0.500 

0.9831 

( 

0.600 

0.9902 

( 

0.700 

0.9948 

( 

0.800 

0.9976 

( 

0.900 

0.9993 

( 

0.950 

0.9998 

( 

0.2628  ,  0.7217  ) 

0.3620  ,  0.7648  ) 

0.4772  ,  0.8107  ) 

0.5483  ,  0.8389  ) 

0.5986  ,  0.8596  ) 

0.6367  ,  0.8759  ) 

0.6669  ,  0.8894  ) 

0.6916  ,  0.9008  ) 

0.7122  ,  0.9106  ) 

0.7298  ,  0.9192  ) 

0.7450  ,  0.9267  ) 

0.7583  ,  0.9334  ) 

0.7701  ,  0.9393  ) 

0.7806  ,  0.9446  ) 

0.7901  ,  0.9494  ) 

0.7987  ,  0.9537  ) 

0.8325  ,  0.9699  ) 

0.8566'  ,  0.9802  ) 

0.8751  ,  0.9869  ) 

0.9026  ,  0.9943  ) 

0.9229  ,  0.9976  ) 

0.9392  ,  0.9991  ) 

0.9532  ,  0.9997  ) 

0.9660  ,  0.9999  ) 

0.9785  ,  1.0000  ) 

0.9856  ,  1.0000  ) 


ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT  LOWER  BOUND  UPPER  BOUND 


(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(0.0151, 

(0.0541, 

(0.1039, 

(0.4507, 

0.6247) 

0.7837) 

0.8585) 

0.9783) 

(0.0043, 

(0.0275, 

(0.0639, 

(0.3725, 

0.4726) 

0.7002) 

0.8035) 

0.9680) 

(0.0443, 

(0.0978, 

(0.1595, 

(0.5310, 

0.7594) 

0.8520) 

0.9020) 

0.9857) 

R  0  C  F  I  T  (JUNE  1993  VERSION)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMAL 
NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  82. 

RESPONSE  DATA: 

CATEGORY  1  2 

ACTUALLY  NEGATIVE  CASES  98.  39. 

ACTUALLY  POSITIVE  CASES  5.  7. 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0336  0.0671  0.0805  0.3423  1.0000 


3  4  5 

2.  5.  5. 

0 .  8 .  62 , 


TPF;  0.0000  0.7561  0.8537  0.8537  0.9390  1.0000 


INITIAL  VALUES  OF  PARAMETERS: 

A=  1.8145  B=  0.5476 

Z(K)=  0.4058  1.4017  1.4979  1.8313 

LOGL=  -205.7691 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  A: 


PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.8307  B=  0.5978 

Z(K)=  0.4095  1.3735  1.4355  1.8921 

LOGL=  -204.5984 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  A 


A 

VARIANCE-COVARIANCE  MATRIX': 
0.0774  0.0364  0.0075  0.0040 

0.0032 

-0.0060 

B 

0.0364 

0.0270 

0.0024 

-0.0042 

-0.0051 

-0.0154 

Z( 

1) 

0.0075 

0.0024 

0.0112 

0.0067 

0.0065 

0.0051 

Z( 

2) 

0.0040 

-0.0042 

0.0067 

0.0202 

0.0199 

0.0195 

Z( 

3) 

0.0032 

-0.0051 

0.0065 

0.0199 

0.0215 

0.0212 

Z( 

4) 

-0.0060 

-0,0154 

0.0051 

0.0195 

0.0212 

0.0394 

A 

CORRELATION  MATRIX: 
1.0000  0.7951  0.2546 

0.1018 

0.0795 

-0.1083 

B 

0.7951 

1.0000 

0.1361 

-0.1777 

-0.2116 

-0.4724 

Z( 

1) 

0.2546 

0.1361 

1.0000 

0.4468 

0.4199 

0.2418 

Z( 

2) 

0.1018 

-0.1777 

0.4468 

1.0000 

0.9569 

0.6916 

Z( 

3) 

0.0795 

-0.2116 

0.4199 

0.9569 

1.0000 

0.7298 

Z( 

4) 

-0.1083 

-0.4724 

0.2418 

0.6916 

0.7298 

1.0000 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 


TPF 


(LOWER  BOUND,  UPPER  BOUND) 


0.005 
0.010 
0.020 
0.030 
0.040 
0.050 
0.060 
0.070 
0.080 
090 
100 
110 
120 
0.130 
0.140 
0.150 
0.200 
0.250 
0.300 


0.6144 

0.6700 

0.7267 

0.7600 

0.7835 

0.8016 

0.8162 

0.8285 

0.8391 

0.8483 

0.8565 

0.8638 

0.8704 

0.8764 

0.8820 

0.8871 

0.9079 

0.9233. 

0.9354 


( 


0.4107 

0.4929 

0.5796 

0.6305 

0.6658 

0.6924 

0.7135 

0.7307 

0.7452 

0.7576 

0.7683 

0.7778 

0.7862 

0.7937 

0.8005 

0.8067 

0.8313 

0.8491 

0.8630 


0.7902 

0.8153 

0.8425 

0.8598 

0.8728 

0.8833 

0.8923 

0.9000 

0.9069 

0.9131 

0.9186 

0.9237 

0.9283 

0.9326 

0.9365 

0.9402 

0.9551 

0.9658 

0.9739 


0.400 

0.9535 

(  0.8843 

9 

0.9847 

) 

0.500 

0.9664 

(  0.9007 

9 

0.9912 

) 

0.600 

0.9763 

(  0.9147 

9 

0.9953 

) 

0.700 

0.9840 

(  0.9274 

9 

0.9977 

) 

0.800 

0.9902 

(  0.9401 

9 

0.9991 

) 

0.900 

0.9953 

(  0.9543 

9 

0.9998 

) 

0.950 

0.9976 

(  0.9637 

9 

0.9999 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT  LOWER  BOUND  UPPER  BOUND 


(  FPF  , 

TPF  .)  ■ 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(0.0292, 

(0.0756, 

(0.0848, 

(0.3411, 

0.7580) 

0.8346) 

0.8437) 

0.9436) 

(0.0113, 

(0.0425, 

(0.0493, 

(0.2687, 

0.6798) 

0.7884) 

0.8005) 

0.9281) 

(0.0664, 

(0.1255, 

(0.1368, 

(0.4199, 

0.8244) 

0.8738) 

0.8802) 

0.9564) 

R  0  C  F  I  T  (JUNE  1993  VERSION)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMAI 
NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  27. 

RESPONSE  DATA: 

CATEGORY  12  3  4  5 

ACTUALLY  NEGATIVE  CASES  87.  40.  6.  12.  4. 

ACTUALLY  POSITIVE  CASES  5.  7.  1.  8.  6. 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0268  0.1074  0.1477  0.4161  1.0000 
TPF:  0.0000  0.2222  0.5185  0.5556  0.8148  1.0000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.1369  B=  0.9553 

Z(K)=  0.2115  1.0466  1.2407  1.9297 

LOGL=  -203.8137 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AF 


PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.1181  B=  0.9444 

Z(K)=  0.2135  1.0427  1.2129  1.9635 

LOGL=  -203.6756 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AR 


VARIANCE-COVARIANCE  MATRIX: 

A  0.1028  0.0501  0.0105  0.0077  0.0063  -0.0052 


B 

0.0501 

0.0503 

0.0036 

-0.0028 

-0.0051 

-0.0218 

Z( 

1) 

0.0105 

0.0036 

0.0107 

0.0066 

0.0061 

0.0040 

Z( 

2) 

0.0077 

-0.0028 

0.0066 

0.0149 

0.0140 

0.0123 

Z( 

3) 

0.0063 

-0.0051 

0.0061 

0.0140 

0.0171 

0.0154 

Z( 

4) 

-0.0052 

-0.0218 

0.0040 

0.0123 

0.0154 

0.0438 

CORRELATION  MATRIX: 

A 

1.0000 

0.6973 

0.3172 

0.1970 

0.1497 

-0.0771 

B 

0.6973 

1.0000 

0.1548 

-0.1016 

-0.1732 

-0.4642 

Z( 

1) 

0.3172 

0.1548 

1.0000 

0.5240 

0.4478 

0.1836 

Z( 

2) 

0.1970 

-0.1016 

0.5240 

1.0000 

0.8780 

0.4818 

Z( 

3) 

0.1497 

-0.1732 

0.4478 

0.8780 

1.0000 

0.5624 

Z( 

4) 

-0.0771 

-0.4642 

0.1836 

0.4,818 

0.5624 

1.0000 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


0.005 

0.0943 

( 

0.010 

0.1402 

( 

0.020 

0.2056 

( 

0.030 

0.2551 

( 

0.040 

0.2961 

( 

0.050 

0.3316 

( 

0.060 

0.3630 

( 

0.070 

0.3913 

( 

0.080 

0.4172 

( 

0.090 

0.4411 

( 

0.100 

0.4632 

( 

0.110 

0.4839 

'  ( 

0.120 

0.5033 

( 

0.130 

0.5217 

( 

0.140 

0.5390 

( 

0.150 

0.5554 

( 

0.200 

0.6268 

( 

0.250 

0.6849 

( 

0.300 

0.7334 

( 

0,400 

0.8104 

( 

0.500 

0.8682 

( 

0.600 

0.9126 

( 

0.700 

0.9466 

( 

0.800 

0.9721 

( 

0.900 

0.9901 

( 

0.950 

0.9962 

( 

0.0161 

f 

0.3129 

) 

0.0346 

/ 

0.3663 

) 

0.0709 

/ 

0.4306 

) 

0.1050 

/ 

0.4746 

) 

0.1368 

/ 

0.5094 

) 

0.1664 

/ 

0.5388 

) 

0.1940 

/ 

0.5645 

) 

0.2197 

f 

0.5876 

) 

0.2439 

r 

0.6087 

) 

0.2665 

r 

0.6281 

) 

0.2878 

/ 

0.6462 

) 

0.3078 

t 

0.6632 

) 

0.3268 

9 

0.6793 

) 

0.3447 

9 

0.6944 

) 

0.3616 

9 

0.7088 

) 

0.3777 

9 

0.7225 

) 

0.4474 

9 

0.7821 

) 

0.5036 

9 

0.8299 

) 

0.5506 

9 

0.8685 

) 

0.6265 

9 

0.9245 

) 

0.6878 

9 

0.9596 

) 

0.7410 

9 

0.9807 

) 

0.7901 

9 

0.9922 

) 

0.8384 

9 

0.9977 

) 

0.8907 

9 

0.9997 

) 

0.9231 

9 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 
(  FPF  ,  TPF  ) 


LOWER  BOUND  UPPER  BOUND 

FPF  ,  TPF  )  (  FPF  ,  TPF  ) 


(0.0248,  0.2308) 


(0.0088,  0.1306) 


(0.0602,  0.3636) 


(0.1126, 

0.4891) 

(0.0709,  0.3939) 

(0.1694, 

0.5850) 

(0.'1485, 

0.5531) 

(0.1000,  0.4633) 

(0.2108, 

0.6402) 

(0.4155, 

0.8203) 

(0.3386,  0.7658) 

(0.4957, 

0.8660) 

R  0  C  F  I  T  (JUNE  1993 

VERSION) 

: 

MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMAI 


NO.  OF  ACTUALLY  NEGATIVE  CASES  = 

149. 

NO.  OF 

ACTUALLY  POSITIVE  CASES  = 

RESPONSE  DATA: 

CATEGORY 

1 

2 

3 

4 

5 

ACTUALLY  NEGATIVE  CASES 

43. 

87. 

19. 

0. 

0. 

ACTUALLY  POSITIVE  CASES 

2. 

24. 

89. 

24. 

11. 

OBSERVED  OPERATING  POINTS: 
FPF:  0.0000  0.0000  0.0000 
TPF:  0.0000  0.0733  0.2333 

0.1275 

0.8267 

0.7114  1,0000 
0.9867  1.0000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.8003  B=  1.0541 

Z(K)=  -0.5571  1.1383  2.6112  2,7112 

LOGL=  -343.1591 

CHI-SQUARE  GOODNESS  OF  FIT  NOT • CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AI 


PROCEDURE  CONVERGES  AFTER  7  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1,7862  B=  0.7355 

Z(K)=  -0.5591  1.1430  3.4209  4.4039 

LOGL=  -311.2273 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AP 


A 

B 

Z(  1) 
Z(  2) 
Z(  3) 
Z(  4) 


VARIANCE-COVARIANCE  MATRIX: 
0.0407  0.0213  0.0064  0.0068 

0.0213  0.0236  0.0032  -0.0046 

0.0064  0.0032  0.0118  0.0036 

0.0068  -0.0046  0.0036  0.0168 

-0.0468  -0.0791  -0.0060  0,0303 

-0.0744  -0,1112  -0.0102  0,0364 


-0.0468  -0.0744 
-0.0791  -0.1112 
-0.0060  -0,0102 
0.0303  0.0364 

0.3152  0.4174 

0.4174  0.5974 


A 

B 

Z(  1) 
Z(  2) 
Z(  3) 
Z(  4) 


CORRELATION  MATRIX: 

1.0000  0.6882  0.2914  0.2582 

0.6882  1.0000  0.1894  -0.2296 

0.2914  0.1894  1.0000  0.2550 

0.2582  -0.2296  0.2550  1.0000 

-0.4137  -0.9163  -0.0979  0.4159 

-0.4775  -0.9358  -0.1216  0.3636 


-0.4137 

-0.9163 

-0.0979 

0.4159 

1.0000 

0.9618 


-0.4775 

-0.9358 

-0.1216 

0.3636 

0.9618 

1.0000 


ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 


BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 


FALSE-POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


0.005 

0.4568 

( 

0.010 

0.5299 

( 

0 . 02  0 

0.6085 

( 

0.030 

0.6564 

( 

0.040 

0.6909 

( 

0.050 

0.7178 

( 

0.060 

0.7397 

( 

0.070 

0.7582 

( 

0.080 

0.7742 

( 

0.090 

0.7882 

( 

0.100 

0.8006 

( 

0.110 

0.8117 

( 

0.120 

0.8217 

( 

0.130 

0.8309 

( 

0.140 

0.8393 

( 

0.150 

0.8471 

( 

0.200 

0.8785 

( 

0.250 

0.9015 

( 

0.300 

0.9194 

( 

0.400 

0.9452 

( 

0.500 

0.9630 

( 

0.600 

0.9757 

( 

0.700 

0.9851 

( 

0.800 

0.9919 

( 

0.900 

0.9968 

( 

0.950 

0.9986 

( 

0.2456 

f 

0.6814 

) 

0.3296 

f 

0.7227 

) 

0.4307 

f 

0.7659 

) 

0.4966 

f 

0.7922 

) 

0.5455 

r 

0.8112 

) 

0.5842 

/ 

0.8264 

) 

0.6160 

/ 

0.8390 

) 

0.6428 

f 

0.8498 

) 

0.6658 

t 

0.8593 

) 

0.6859 

f 

0.8677 

) 

0.7037 

r 

0.8754 

) 

0.7194 

r 

0.8824 

) 

0.7336 

f 

0.8888 

) 

0.7464 

/ 

0.8948 

) 

0.7580' 

/ 

0.9003 

) 

0.7686 

f 

0.9055 

) 

0.8106 

/ 

0.9271 

) 

0.8402 

9 

0.9436 

) 

0.8626 

9 

0.9563 

) 

0.8949 

9 

0.9743 

) 

0.9179 

9 

0.9854 

) 

0.9359 

0.9923 

) 

0.9510 

9 

0.9964 

) 

0.9646 

9 

0.9987 

) 

0.9778 

9 

0.9997 

) 

0.9853 

9 

0.9999 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT  LOWER  BOUND  UPPER  BOUND 


(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(0.0000, 

(0.0003, 

(0.1265, 

(0.7120, 

0.0732) 

0.2328) 

0.8278) 

0.9860) 

(0.0000, 

(0.0000, 

(0.0812, 

(0.6355, 

0.0051) 

0.0619) 

0.7760) 

0.9794) 

(0.0019, 

(0.0102, 

(0.1870, 

(0.7799, 

0.3675) 

0.5317) 

0.8713) 

0.9907) 

R  0  C  F  I  T  (JUNE  1993  VERSION)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
0-F  A.  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  9,  Mass  Question 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNO 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  125.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  5 

RESPONSE  DATA: 

CATEGORY 

ACTUALLY  -NEGATIVE  CASES 
ACTUALLY- POSITIVE  CASES 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0000  0.0000  0.1040  0.3040  1.0000 
TPF:  0.0000  0.4314  0.6078  0.8039  0.9216  1.0000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.7240  B=  0.6484 

Z(K)=  0.5125  1.2592  2.5525  2.6525 

LOGL=  -186.1320 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIE. 


PROCEDURE  CONVERGES  AFTER  8  ITERATIONS. 


1  2  3  4  5 

87.  25.  13.  0.  0. 

4-  6.  10.  9.  22. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.5837  B=  0.4840 

Z(K)=  0.5062  1.3070  2.7322  3.6470 

LOGL=  -175.6608 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIEi 


A 

B 

Z(  1) 
Z(  2) 
Z(  3) 
Z(  4) 


VARIANCE -COVARIANCE  MATRIX : 


0.0885  0.0335  0.0074 

0.0335  0.0230  0.0020 

0.0074  0.0020  0.0138 

0.0054  -0.0021  0.0091 

-0.0349  -0.0488  0.0041 

-0.0885  -0.0949  0.0004 


0.0054  -0.0349  -0.0885 
-0.0021  -0.0488  -0.0949 
0.0091  0.0041  0.0004 

0.0226  0.0227  0.0268 

0.0227  0.2101  0.2978 

0.0268  0.2978  0.5556 


A 

B 

Z(  1) 
Z(  2) 
Z(  3) 
Z{  4) 


CORRELATION  MATRIX: 
1.0000  0.7424  0.2112 

0.7424  1.0000  0.1114 

0.2112  0.1114  1.0000 

0.1205  -0.0941  0.5152 

-0.2558  -0.7011  0.0759 

-0.3991  -0.8390  0.0042 


0.1205  -0.2558  -0.3991 
-0.0941  -0.7011  -0.8390 
0.5152  0.0759  0.0042 

1.0000  0.3294  0.2388 

0.3294  1.0000  0.8716 

0.2388  0.8716  1.0000 


AREA  =  0.9230  STD.  DEV. (AREA)  =  0.0307 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE -POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF  TPF 


(LOWER  BOUND,  UPPER  BOUND) 


0.005 

0.6318 

0.01-0 

0,6763 

0.020 

0.7222 

0.030 

0.7496 

0.040 

0.7692 

0.050 

0.7845 

0.060 

0.7970 

0.070 

0.8076 

0.080 

0.8169 

0.090 

0.8250 

0.100 

0.8323 

0.110 

0.8389 

0.120 

.  0.8449 

0.130 

0.8505 

0.140 

•  0.8556 

0.150  ' 

■  0.8604 

0.200 

0.8803 

0.250 

0.8957 

0,300 

0.9082 

0.400 

0.9280 

0.500 

0.9434 

0.600 

0.9560 

0.700 

0.9669 

0.800 

0.9768 

0.900 

0.9862 

0.950 

0.9913 

(  0.4299 
(  0.4955 
(  0.5636 
(  0.6035 
(  0.6315 
(  0.6527 
(  0.6698 
(  0.6839 
(  0.6959 
(  0.7063 
(  0.7154 
(  0.7236 
(  0.7309 
(  0.7375 
(  0.7436 
(  0.7492 
(  .  0.7719 
(  0.7890 
(  0.8027 
(  0.8242 
(  0.8415 
(  0.8566 
(  0.8709 
(  0.8857 
(  0.9034 
(  0.9160 


0.8024  ) 
0.8228  ) 
0.8458  ) 
0.8608  ) 
0.8721  ) 
0.8814  ) 
0.8893  ) 
0.8961  ) 
0.9022  ) 
0.9077  ) 
0.9127  ) 
0.9172  ) 
0.9214  ) 
0.9252  ) 
0.9288  ) 
0.9322  ) 
0.9460  ) 
0.9565  ) 
0.9648  ) 
0.9768  ) 
0.9849  ) 
0.9905  ) 
0.9945  ) 
0.9973  ) 
0.9991  ) 
0.9996  ) 


ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT  LOWER  BOUND  UPPER  BOUND 


(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(0.0001, 

(0.0031, 

(0.0956, 

(0.3064, 

0.4280) 

0.6030) 

0.8292) 

0.9097) 

(0.0000, 

(0.0001, 

(0.0546, 

(0.2308, 

0.1871) 

0.4311) 

0.7906) 

0.8902) 

(0.0144, 

(0.0333, 

(0.1557, 

(0.3912, 

0.7004) 

0.7568) 

0.8630) 

0.9265) 

R  0  C  F  I  T  (JUNE  1993  VERSION)  : 


MAXIMUM  LIKELI'HOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  9,  MicroCalcifications 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORi 


NO.  OF  ACTUALLY  NEGATIVE  CASES  = 

125. 

NO.  OF 

ACTUALLY  POSITIVE  CASES 

RESPONSE  DATA: 

CATEGORY 

1 

2 

3 

4 

5 

ACTUALLY  NEGATIVE  CASES 

120. 

4. 

0. 

0. 

1 . 

ACTUALLY  POSITIVE  CASES 

23. 

0* 

2. 

5. 

35. 

OBSERVED  OPERATING  POINTS: 
FPF:  0.0000  0.0080  0.0080 

0.0080 

0.0400 

1.0000 

TPF:  0.0000  0.5385  0.6154  0.6462  0.6462  1.0000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.3348  ,  B=  0.4725 

Z(K)=  1.7511  2.2093  2.3093  2.4093 

LOGL=  -98.6914 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES 


PROCEDURE  CONVERGES  AFTER  8  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  0.8911  B=  0.2887 

Z(K)=  1.7590  2.0535  2.2508  2.7339 

LOGL=  -94.1625  , 

CHI -SQUARE. GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES 


VARIANCE-COVARIANCE  MATRIX: 

A  0.1439  0.0582  0.0187  -0.0051  -0.0258  -0.0947 
B  0.0582  0.0291  0.0038  -0.0093  -0.0206  -0.0582 
Z(  1)  0.0187  0.0038  0.0419  0.0377  0.0351  0.0291 
Z(  2)  -0.0051  -0.0093  0.0377  0.0578  0.0596  0.0693 
Z(  3)  -0.0258  -0.0206  0.0351  0.0596  0.0815  0.1048 
Z(  4)  -0.0947  -0.0582  0.0291  0.0693  0.1048  0.2247 


CORRELATION  MATRIX: 

A  1.0000  0.9004  0.2413  -0.0564  -0.2380  -0.5266 
B  0.9004  1.0000  0.1084  -0.2260  -0.4226  -0.7202 
Z(  1)  0.2413  0.1084  1.0000  0.7662  0.6009  0.2999 
Z(  2)  -0.0564  -0.2260  0.7662  1.0000  0.8682  0.6084 
Z(  3)  -0.2380  -0.4226  0.6009  0.8682  1.0000  0.7746 
Z(  4)  -0.5266  -0.7202  0.2999  0.6084  0.7746  1.0000 


AREA  =  0.8040  STD.  DEV. (AREA)  =  0.0912 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF- 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.5585 

( 

0.4096  , 

0.6995 

) 

0.010 

0.5868 

( 

0.4516  , 

0.7123 

) 

0.020 

■  0.6171 

( 

0.4897  , 

0.7330 

) 

0.030 

0.6360 

( 

0.5087  , 

0.7498 

) 

0.040 

0.6501 

( 

0.5204  , 

0.7642 

) 

0.050 

0.6613 

( 

0.5284  , 

0.7767 

) 

0.060 

0.6708 

( 

0.5341  , 

0.7877 

) 

0.070 

0.6790 

( 

0.5384  , 

0.7976 

) 

0.080 

0.6863 

( 

0.5418  , 

0.8066 

) 

0.090 

0.6928 

( 

0.5446  , 

0.8148 

) 

0.100 

0.6988 

( 

0.5468  , 

0.8224 

) 

0.110 

0.7043 

( 

0.5486  , 

0.8294 

) 

0.120 

0.7094 

( 

0.5501  , 

0.8358 

) 

0.130 

0.7142 

( 

0.5515  , 

0.8419 

) 

0.140 

0.7188 

( 

0.5526  , 

0.8476 

) 

0.150 

.0.7230 

( 

0.5535  , 

0.8529 

) 

0.200 

0.7415 

( 

0.5567  , 

0.8756 

) 

0.250 

0.7569 

( 

0.5584  , 

0.3936 

) 

0.300 

0.7703 

( 

0.5592  , 

0.9083 

) 

0.400  0.7933  (  0.5595  ,  0.9314  ) 
0.500  0.8136  (  0.5587  ,  0.9489  ) 
0.600  0.8325  (  0.5571  ,  0.9628  ) 
0.70.0  0.8514  (  0.5549  ,  0.9742  ) 
0.800  0.8716  (  0.5517  ,  0.9837  ) 
0.900  0.8964  (  0.5465  ,  0.9919  ) 
0.950  O'. 9140  ■  (  0.5417  ,  0.9957  ) 


ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING- POINT  LOWER  BOUND  UPPER  BOUND 


(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  )  , 

(  FPF  , 

TPF  ) 

(0.0031, 

(0.0122, 

(0.0200, 

(0.0393, 

0.5405) 

0.5953) 

0.6172) 

0.6492) 

(0.0001, 

(0.0025, 

(0.0058, 

(0.0154, 

0.4339) 

0.5317) 

0.5644) 

0.6054) 

(0.0355, 

(0.0454, 

(0.0568, 

(0.0872, 

0.6443) 

0.6564) 

0.6679) 

0.6911) 

R  0  C  F  I  T  (JUNE  1993  VERSION)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DA.TA 


DATA  DESCRIPTION:  Reader  9,  FAS/AD 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNOR 
NO.  OF  ACTUALLY  NEGATIVE  CASES  =  125.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  27 

RESPONSE  DATA: 

CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0080  0.0080  0.0400  0.1200  1.0000 
TPF:  0.0000  0.1111  0.3333  0.4074  0.4815  1.0000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  0.9060  B=  0.7267 

Z(K)=  1.1751  1.7511  2.3093  2.4093 

LOGL=  -104.1775 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES 


PROCEDURE  CONVERGES  AFTER  9  ITERATIONS 


FINAL  VALUES  OF  PARAMETERS: 

A=  0.7393  B=  0.6325 

Z(K)=  1.1833  1.6786  2.1371  2.9004 

LOGL=  -96.8623 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES 


VARIANCE-COVARIANCE  MATRIX: 

0.2046  0.'0948  0.0185  0.0021  -0.0236  -0.1023 


1  2  3  4  5 

110.  10.  4.  0.  1. 

14.  2.  2.  6.  3. 


B  0.0-948  0.0614  0.0043  -0.0083  -0.0279  -0.0864 

Z(  1)  0.0185  0.0043  0.0213  0.0172  0.0144  0.0097 

Z(  2)  0.0021  -0.0083  0.0172  0.0338  0.0328  0.0381 

Z(  3)  -0.0236  -0.0279  0.0144  0.0328  0.0676  0.0869 

Z(  4)  -0.1023  -0.0864  0.0097  0.0381  0.0869  0.2634 


CORRELATION  MATRIX; 

A  1.0000  0.8458  0.2800  0.0251  -0.2005  -0.4407 

B  0.8458  1,0000  0.1192  -0.1824  -0.4333  -0.6794 

Z(  1)  0.2800  0.1192  1.0000  0.6440  0.3785  0.1302 

Z(  2)  0.0251  -0.1824  0.6440  1.0000  0.6866  0.4042 

Z(  3)  -0.2005  -0.4333  0.3785  0.6866  1.0000  0.6512 


Z(  4)  -0.4407  -0.6794  0.1302  0.4042  0.6512  1.0000 


-AREA  =  0.7339  STD.  DEV.  (AREA)  =  0.1068 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON- ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 

0.005  0.1867  (  0.0572  ,  0.4202  ) 


0.010  0.2319  ( 
0.020  0.2877  ( 
0.030  0.3261  ( 
0.040  0.3563  ( 
0.050  0.3816  ( 
0.060  0.4035  ( 
0.070  0.4229  ( 
0.080  0.4405  ( 
0.090  0.4566  .  ( 
0,100  0.4715  ( 
0.110  0.4854  ( 
0.120  0.4984  ( 
0.130  0.5107  ( 
0.140  0.5223  ( 
0.150  0.5334  ( 
0.200  0.5820  ■  ( 
0.250  0.6228  ( 
0.300  .  0.6583  ( 
0.400  0.7188  ( 
0.500  0.7701  ( 
0.600  0.8157  ( 
0.700  0.8579  ( 
0.800  '  0.8982  ( 
0.900  0.9394  ( 
0.950  0.9625  ( 


0.0903  ,  0.4499  ) 

0.1370  ,  ■  0.4895  ) 

0.1708  ,  0.5199  ) 

0.1972  ,  0.5459  ) 

0.2186  ,  0.5691  ) 

0.2366  ,  0.5904  ) 

0.2519  ,  0.6102  ) 

0.2652  ,  0.6286  ) 

0.2769  ,  0.6459  ) 

0.2873  ,  0.6622  ) 

0.'2966  ,  0.6776  ) 

0.3051  ,  0.6921  ) 

0.3128  ,  0.7059  ) 

0.3199  ,  0.7190  ) 

0.3264  ,  0.7315  ) 

•0.3533  ,  0.7854  ) 

0.3739  ,  0.8282  ) 

0.3908  ,  0.8628  ) 

0.4183  ,  0.9138  ) 

0.4414  ,  0.9480  ) 

0.4628  ,  0.9707  ) 

6.4843  ,  0.9854  ) 

0.5081  ,  0.9942  ) 

0.5393  ,  0.9987  .  ) 

0.5638  ,  0.9997  ) 


ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT  LOWER  BOUND  UPPER  BOUND 

(  FPF  ,  TPF  )  •  (  FPF  ,  TPF  )  (  FPF  ,  TPF  ) 


(0.0019,  0.1367) 


(0.0000,  0.0417) 


(0.0291,  0.3231) 


(0.0163,  0.2701)  (0.0041,  0.1749)  (0.0518,  0.3859) 

(0.0466,  0.3735)  (0.0207,  0.2911)  (0.0937,  0.4623) 

(0.1183,  0.4963)  (0.0709,  0.4247)  (0.1847,  0.5681) 

R  0  C  F  I  T  (JUNE  1993  VERSION)  ; 

MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  9,  Benign  or  Malignant 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNOF 
NO.  OF  ACTUALLY  NEGATIVE  CASES  =  125.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  12t 

RESPONSE- DATA: 

CATEGORY  1 

ACTUALLY  NEGATIVE  CASES  74. 

ACTUALLY  POSITIVE  CASES  14. 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0000  0.0000  0.0480  0.4080  1.0000 
TPF:  0.0000  0.0960  0.1760  0.6080  0.8880  1.0000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.5928  B=  1.0021 

Z(K)=  0.2323  1.6649  2.5525  2.6525 

LOGL=  -291.5787 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES 


PROCEDURE  CONVERGES  AFTER  6  ITERATIONS. 


2  3  4  5 

45 .  6 .  0 .  0 . 

35.  54.  10.  12. 


FINAL  VALUES  OF  PAIU^METERS: 

A=  1.3650  B=  0.6517 

Z(K)=  0.2320  1.6703  3.5245  4.0979 

LOGL=  -276.9216 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES 


VARIANCE-COVARIANCE  MATRIX: 

A  0.0333  0.0153  0.0090  0.0013  -0.0356  -0.0483 
B  0.0153  0.0162  0.0030  -0.0129  -0.0612  -0.0759 
Z(  1)  0.0090  0.0030  0.0128  0.0060  -0.0025  -0.0052 
Z(  2)  0.0013  -0.0129  0.0060  0.0358  0.0715  0.0829 
Z(  3)  -0.0356  -0.0612  -0.0025  0.0715  0.2948  0.3453 
Z(  4)  -0.0483  -0.0759  -0.0052  0.0829  0.3453  0.4386 


CORRELATION  MATRIX: 

A  1.0000  0.6584  0.4356 
B  0.6584  1.0000  0.2100 
Z(  1)  0.4356  0.2100  1.0000 
Z(  2)  0.0388  -0.5363  0.2790 
Z(  3)  -0.3594  -0.8854  -0.0408 
Z(  4)  -0.4000  -0.9004  -0.0689 


0.0388  -0.3594  -0.4000 
-0.5363  -0.8854  -0.9004 
0.2790  -0.0408  -0.0689 
1.0000  0.6956  0.6613 

0.6956  1.0000  0.9602 

0.6613  0.9602  1.0000 


AREA  =  0.8736  STD.  DEV.  (AREA)  =  0.0249 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE -POSITIVE  FRACTION  AT  EACH  SPECIFIED 


FALSE-POSITIVE  FRACTION: 


FPF.  TPF 


(LOWER  BOUND,  UPPER  BOUND) 


0.005 

0.3767 

0.010 

0.4398 

0.020 

0.5104 

0.030 

0.5552 

0.040 

0.5885 

0.050 

0.6151 

0.060 

0.6374 

0.070 

0.6565 

0.080 

0,6733 

0.090 

0.6883 

0.100 

0.7018 

0.110 

.  0.7141 

0.120 

•  0.7254 

0.130 

0.7359 

0.140 

0.7456 

0.150 

0.7547 

0.200 

0.7929 

0.250 

0.8227 

0.300 

0.8470 

0.400 

0.8850 

0.500 

0.9139 

0.600 

0.9370 

0.700 

0.9560 

0.800 

0.9721 

0.900 

0.9861 

0.950 

0.9926 

(  0.2113 
(  0.2780 
(  0.3595 
(  0.4139 
(  0.4553 
(  0.4887 
(  0.5168 
(  0.5410 
(  0.5621 
(  0.5809 
(  0.5978 
(  0.6130 
(  0.6270 
(  0.6397 
(  ■  0.6515 
(  0.6625 
(  0.7075 
(  0.7415 
(  0.7687 
(  0.8107 
(  0.8431 
(  0.8703 
(  0.8947 
(  0.9182 
C  0.9433 
(  0.9589 


0.5690  ) 
0.6125  ) 
0.6599  ) 
0.6898  ) 
0.7122  ) 
0.7303  ) 
0.7456  ) 
0.7590  ) 
0.7709  ) 
0.7817  ) 
0.7915  ) 
0.8006  ) 
0.8090  ) 
0.8170  ) 
0.8244  ) 
0.8314  ) 
0.8615  ) 
0.8855  ) 
0.9053  ) 
0.9357  ) 
0.9575  ) 
0.9733  ) 
0.9847  ) 
0.9925  ) 
0.9976  ) 
0.9991  ) 


ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 
(  FPF  ,  TPF  ) 


LOWER  BOUND  UPPER  BOUND 

FPF  ,  TPF  )  (  FPF  ,  TPF  ) 


(0.0000,  0.0958) 
(0.0002,  0.1756) 
(0.0474,  0.6089) 
(0.4083,  0.8876) 


(0.0000,  0.0157) 
(0.0000,  0.0520) 
(0.0206,  0.5138) 
(0.3250,  0.8575) 


(0.0026,  0.3228) 
(0.0069,  0.4058) 
(0.0969,  0.6978) 
(0.4959,  0.9128) 


R  0  C  F  I  T  (JUNE  1993  VERSION)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF-A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Read6r  10,  Mass  Question 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMA] 


NO.  OF  ACTUALLY  NEGATIVE  CASES  = 

100. 

NO.  OF 

ACTUALLY  POSITIVE  CASES  = 

RESPONSE  DATA: 

CATEGORY 

1 

2 

3 

4 

5 

ACTUALLY  NEGATIVE  CASES 

94. 

4. 

1. 

0. 

1. 

ACTUALLY  POSITIVE  CASES 

14.  - 

3. 

3. 

4. 

18. 

OBSERVED  OPERATING  POINTS: 
FPF:  0.0000  0.0100  0.0100 
TPF:  0.0000  0.4286  0.5238 

0.0200 

0.5952 

0.0600 

0.6667 

1.0000 

1.0000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.5622  B=  0.6980 

Z(K)=  1.5551  2.0542  2.2268  2.3268 

LOGL=  -87.2521 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AI 


PROCEDURE  CONVERGES  AFTER  5  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.4407  B=  0.6401 

Z(K)=  1.5592  1.9417  2.2197  2.5170 

LOGL=  -84.9281 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AF 


A 

B 

Z(  1) 
Z(  2) 
Z(  3) 
Z(  4) 


VARIANCE-COVARIANCE  MATRIX: 


0.3827  0.1857 
0.1857  0.1039 
0.0387  0.0084 
0.0155  -0.0248 
0.0682  -0.0567 
0.1359  -0.0972 


0.0387  -0.0155 
0.0084-0.0248 
0.0400  0.0346 
0.0346  0.0585 
0.0311  0.0639 
0.0273  0.0722 


-0.0682  -0.1359 
-0.0567  -0.0972 
0.0311  0.0273 

0.0639  0.0722 

0.0962  0.1157 

0.1157  0.1722 


A 

B 

Z(  1) 
Z(  2) 
Z(  3) 
Z(  4) 


CORRELATION  MATRIX: 

0.3127  -0.1038 
0.1307  -0.3180 


1.0000  0.9310 
0.9310  1.0000 
0.3127  0.1307 
0.1038  -0.3180 
0.3552  -0.5666 
0.5292  -0.7266 


1.0000  0.7158 
0.7158  1.0000 
0.5011  0.8521 
0.3296  0.7189 


-0.3552  -0..5292 
-0.5666  -0.7266 
0.5011  0.3296 

0.8521  0.7189 

1.0000  0.8989 

0.8989  1.0000 


AREA  =  0.8875 


STD.  DEV. (AREA)  =  0.0691 


ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 


TPF 


(LOWER  BOUND,  UPPER  BOUND) 


0 

.005 

0 

.4174 

( 

0 

.010  ■ 

0 

.4806 

( 

0 

.020 

0 

.5500 

( 

0 

.030 

0 

.5935 

( 

0 

.040 

0, 

.6254 

( 

0 

.050 

0, 

.6508 

( 

0, 

.060 

0, 

.6719 

( 

0 

.070 

0. 

.6900  . 

( 

0, 

.080 

0. 

.7058 

( 

0, 

.090 

0. 

.7198 

( 

0. 

.100 

0. 

.7324 

( 

0, 

.110 

0. 

.7439 

( 

0, 

.120 

0, 

.7544 

( 

0. 

.130 

0. 

.7641  . 

( 

0. 

.140. 

0-. 

.-7731 

( 

0. 

.150 

'0. 

,7815 

( 

0, 

.200 

0. 

,8165 

( 

0, 

.250 

0. 

,8435 

( 

0. 

.300 

0. 

,8655 

( 

0. 

.400 

0. 

,8995 

( 

0. 

,500 

0. 

9252 

( 

0. 

,600 

0. 

9455 

( 

0. 

,700 

0. 

9621 

( 

0. 

,800 

0. 

9761 

( 

0. 

900 

0. 

9881 

( 

0. 

950 

0. 

9937 

( 

0.1907  ,  0.6767  ) 
0.2717  ,  0.6951  ) 
0.3638  ,  0.7256  ) 
0.4167  ,  0.7528  ) 
0.4508  ,  0.7773  ) 
0.4744  ,  0.7993  ) 
0.4916  ,  0.8190  ) 
0.5046  ,  0.8365  ) 
0.5148  ,  0.8520  ) 
0.5230  ,  0.8658  ) 
0.5297  ,  0.8782  ) 
0.5353  ,  0.8892  ) 
0.5402  ,  0.8990  ) 
0.5443  ,  0.9079  ) 
0.5480  ,  0.9159  ) 
0.5512  ,  0.9230  ) 
0.5630  ,  0.9501  ) 
0.5708  ,  0.9671  ) 
0.5765  ,  0.9782  ) 
0.5845  ,  0.9905  ) 
0.5902  ,  0.9960  ) 
0.5948  ,  0.9985  ) 
0.5989  ,  0.9995  ) 
0.6029  ,  0.9999  ) 
0.6074  ,  1.0000  ) 
0.6105  ,  1.0000  ) 


ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS; 


EXPECTED  OPERATING  POINT  LOWER  BOUND  UPPER  BOUND 


(  FPF  , 

TPF  ) 

.  (  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(0.0059, 

(0.0132, 

(0.0261, 

(0.0595, 

0.4323) 

0.5079) 

0.5784) 

0.6710) 

(0.0004, 

(0.0023, 

(0.0078, 

(0.0255, 

0.2447) 

0.3559) 

0.4579) 

0.5760) 

(0.0442, 

(0.0535, 

(0.0711, 

(0.1215, 

0.6369) 

0.6587) 

0.6919) 

0.7560) 

R  0  C  F  I  T  (JUNE  1993  VERSION)  ; 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  10,  MicroCalcifications 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMA 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  100.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  51. 

RESPONSE  DATA; 

CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0400  0.0500  0.0900  1.0000 


1  2  3  4  5 

91 .  4 .  0 .  1 .  4 . 

18.  4.  0.  5.  24. 


TPF:  0.0000  0.4706  0.5686  0.6471  1.0000 


INITIAL  VALUES  OF  PARAMETERS: 

A=  1.7439  8=  1.0039 

Z(K)=  1.3410  1.6452  1.7511 

LOGL=  -99.3411 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  A1 


PROCEDURE  CONVERGES.  AFTER  4  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.7844  B=  1.0437 

Z(K)=  1.3425  1.5801  1.7761 

LOGL=  -98.0825 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  A] 


VARIANCE-COVARIANCE  MATRIX: 


A 

0.6934 

0.4292 

0.0497 

-0.0069 

-0.0612 

B 

0.4292 

0.2914 

0.0129 

-0.0273 

-0.0655 

Z( 

1) 

0.0497 

0.0129 

0.0311 

0.0281 

0.0258 

Z( 

2) 

”0.0069 

-0.0273 

0.0281 

0.0370 

0.0395 

Z( 

3) 

-0.0612 

-0.0655 

0.0258 

0.0395 

0.0529 

A 

CORRELATION  MATRIX: 
1.0000  0.9547  0.3385 

-0.0433 

-0.3193 

B 

0.9547 

1.0000 

0.1350 

.'-0.2  628 

-0.5272 

Z( 

1) 

0.3385 

0.1350 

1.0000 

0.8281 

0.6350 

Z( 

2) 

-0.0433 

-0.2628 

0.8281 

1.0000 

0.8929 

Z( 

3) 

-0.3193 

-0.5272 

0.6350 

0.8929 

1.0000 

AREA  =  0.8915  .  STD.  DEV. (AREA)  =  0.0515 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.1829 

( 

0.0150 

r 

0.6407 

) 

0.010 

0.2598 

( 

0.0475 

r 

0.6488 

) 

0.020 

0.3596 

( 

0.1264 

f 

0.6644 

) 

0.030 

0.4290 

( 

0.2035 

f 

0.6813 

) 

0.040 

0'.4828 

( 

0.2705 

r 

0.7002 

) 

0.050 

0.5268 

( 

0.3258 

/ 

0.7211 

) 

0.060 

0.5641 

( 

0.3699 

r 

0.7437 

) 

0.070 

0.5963 

( 

0.4045 

/ 

0.7672 

) 

0.080 

0.6246 

( 

0.4313 

r 

0.7906 

) 

0.090 

0.6498 

( 

0.4521 

f 

0.8133 

) 

0.100 

0.6725 

( 

0.4684 

r 

0.8346 

) 

0.110 

0.6929 

( 

0.4813 

r 

0.8543 

) 

0.120 

0.7116 

( 

0.4917 

r 

0.8722 

) 

0.130 

0.7287 

( 

0.5002 

0.8882 

) 

0.140 

0.7444 

( 

0.5072 

f 

0.9025 

) 

0.150 

0.7589 

( 

0.5131 

t 

0.9151 

) 

0.200 

0.8176 

( 

0.5321 

f 

0.9584 

) 

0.250 

0.8601 

( 

0.5423 

f 

0.9801 

) 

0.300 

0.8921 

( 

0.5487 

r 

0.9907 

) 

0.400 

0.9358 

( 

0.5563 

r 

0.9981 

) 

0.500 

0.9628 

( 

0.5605 

f 

0.9997 

) 

0.600 

0.9797 

(  0.5632  , 

1.0000 

) 

0.700 

0.9901 

(  0.5650  , 

1.0000 

) 

o.aoo 

0.9961 

(  0.5661  , 

1.0000 

) 

0.900  - 

0.9991 

(  0.5665  , 

1.0000 

) 

0.950 

0.9998 

(  0.5663  , 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT  .  LOWER  BOUND  UPPER  BOUND 

(  FPF  ,  .  TPF  .)  (  FPF  ,  TPF  )  (  FPF  ,  TPF  ) 

(0.0379,  0.4724)  (0.0130,  0.2946)  .  (0.0926,  0.6559) 

(0.0570,  0.5538)  (0.0252,  0.3981)  (0.1145,  0.7016) 

(0.0897,  0.6492)  (0.0457,  0.5090)  (0.1594,  0.7716) 

1  R  0  C  F  I  T  (JUNE  1993  VERSION)  : 

MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  10,  FAS/AD 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMf 
NO.  OF  ACTUALLY  NEGATIVE  CASES  =  100.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  23. 

RESPONSE  DATA: 

CATEGORY  12345 

ACTUALLY  NEGATIVE  CASES  96.-  2.  1.  1.  0. 

ACTUALLY  POSITIVE  CASES  20.  2.  0.  0.  1. 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0000  0.0100  0.0200  0.0400  1.0000 
TPF:  0.0000  0.0435  0.0435  0.0435  0.1304  1.0000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  0.0057  B=  0.6871 

Z(K)=  1.7511  2.0542  2.3268  2.5762 

LOGL=  -34.0185 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  A 


PROCEDURE  CONVERGES  AFTER  7  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  0.1095  B=  0.7126 

Z(K)=  1.7463  2.1891  2.3811  2.7516 

LOGL=  -33.5911 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  A 


VARIANCE-COVARIANCE  MATRIX: 

A  1.2379  0.5909  0.0566  -0.0612-0.1250  -0.2807 
B  0.5909  0.3157  0.0114  -0.0532  -0.0884  -0.1745 
Z(  1)  0.0566  0.0114  0.0514  0.0437  0.0408  0.0356 
Z(  2)  -0.0612  -0.0532  0.0437  0.0968  0.0986  0.1071 


Z(  3)  -0.1250  -0.0884  0.0408  0.0986  0.1412  0.1559 
Z(  4)  -0.2807  -0.1745  0.0356  0.1071  0.1559  0.3118 


CORRELATION  MATRIX: 

A  .  1.0000  0.9453  0.2243  -0.1767  -0.2989  -0.4518 
B  0.9453  1.0000  0.0898  -0.3045  -0.4188  -0.5560 
Z(  1)  0.2243  0.0898  1.0000  0.6202  0.4794  0.2813 
Z(  2)  -0.1767  -0.3045  0.6202  1.0000  0.8436  0.6162 
Z(  3)  -0.2989  -0.4188  0.4794  0.8436  1.0000  0.7429 
Z(  4)  -0.4518  -0.5560  0.2813  0.6162  0.7429  1.0000 


AREA  =  0,5355  STD.  DEV. (AREA)  =  0.3512 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


0 

.005 

0 

.0421 

( 

0 

.010 

0 

.0607 

( 

0 

.020 

0 

.0878 

( 

0 

.030 

0 

.1091 

( 

0 

.040 

0 

.1275 

( 

0 

.050 

0 

.1439 

( 

0 

.060 

0 

.1590 

( 

0 

.070 

0 

.1730 

( 

0 

.080 

0, 

.1862 

( 

0, 

.090 

0, 

.1987 

( 

0, 

.100 

0, 

.2107 

( 

0. 

.110 

0. 

.2222 

( 

0, 

.120 

0. 

.2333 

.  ( 

0, 

.130 

0. 

.2441 

( 

0, 

.140 

0. 

.2545 

( 

0. 

.150 

0, 

.2646 

( 

0, 

.200 

0. 

.3120 

( 

0, 

,250 

0. 

,3553 

( 

0. 

,300 

0, 

,3959 

( 

0. 

.400 

0. 

,4718 

( 

0. 

,500 

0. 

5436 

( 

0. 

600 

0. 

6140 

( 

0. 

700 

0. 

6854 

( 

0. 

800 

0. 

7609 

( 

0. 

900 

0. 

8468 

( 

0. 

950 

0. 

9001 

( 

0.0027  ,  0.2502  ) 
0.0078  ,  0.2487  ) 
0.0181  ,  0.2692  ) 
0.0260  ,  0.3017  ) 
0.0313  ,  0.3392  ) 
0.0346  ,  0.3787  ) 
0.0366  ,  0.4182  ) 
0.0378  ,  0.4570  ) 
0.0384  ,  0.4942  ) 
0.0386  ,  0.5298  ) 
0.0386  ,  0.5634  ) 
0.0334  ,  0.5951  ) 
0.0380  ,  0.6249  ) 
0.0376  ,  0.6528  ) 
0.0371  ,  0.6789  ) 
0.0366  ,  0.7033  ) 
0.0337  ,  0.8021  ) 
0.0308  ,  0.8703  ) 
0.0281  ,  0.9165  ) 
0.0233  ,  0.9678  ) 
0.0192  ,  0.9890  ) 
0.0155  ,  0.9969  ) 
0.0122  ,  0.9993  ) 
0.0091  ,  0.9999  ) 
0.0058  ,  1.0000  ) 
0.0040  ,  1.0000  ) 


ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT  LOWER  BOUND  UPPER  BOUND 


(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(0.0030, 

(0.0086, 

(0.0143, 

(0.040-4, 

0.0321) 

0.0562) 

0.0735) 

0.1282) 

(0.0001, 

(0.0009, 

(0.0026, 

(0.0142, 

0.0043) 

0.0173) 

0.0297) 

0.0733i 

(0.0487, 

(0.0500, 

■  (0.0571, 
(0.0965, 

0.1420) 

0.1440) 

0.1548) 

0.2066) 

R  0  C  F  I  T  (JUNE  1993  VERSION)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF’  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  10,  Benign  or  Malignant 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMA 
NO.  OF  ACTUALLY  NEGATIVE  CASES  =  100.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  100. 

RESPONSE  DATA: 

CATEGORY  1  2  3  4  5 

ACTUALLY  NEGATIVE  CASES  91.  6.  2.  1.  0. 

ACTUALLY  POSITIVE  CASES  37.-22.  20.  6.  15 ! 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0000  0.0100  0.0300  0.0900  1.0000 
TPF:  0.0000  0.1500  0.2100  0.4100  0.6300  1.0000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.8597  B=  1.1295 

Z(K)=  1.3410  1.8812  2.3268  2.5762 

LOGL=  -186.6256 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  A- 


PROCEDURE  CONVERGES  AFTER  5  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.6595  B=  0.9936 

Z(K)=  1.3387  1.8972  2.4656  2.7230 

LOGL=  -186.2753 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  A] 


VARIANCE-COVARIANCE  MATRIX : 

A  0.3016  0.1713  0..0444  -0.0289  -0.1175  -0.1599 
B  0.1713  0.1138  0.0102-0.0408  -0.1014  -0.1302 
Z(  1)  0.0444  0.0102  0.0310  0.0251  0.0193  0.0167 
Z(  2)  -0.0289  -0.0408  0.0251  0.0514  0.0715  0.0813 
Z(  3)  -0.1175  -0.1014  0.0193  0.0715  0.1364  0.1605 


Z(  4)  -0.1599  -0.1302,  0.0167  0.0813  0.1605  0.2006 


A 

B 

Z(  1) 
Z(  2) 
Z(  3) 
Z(  4) 


CORRELATION  MATRIX: 

1.0000  0.9249  0.4596  -0.2320  -0.5793  -0.-6499 

0.9249  1.0000  0.1723  -0.5331  -0.8142  -0.8616 

0.4596  0.1723  1.0000  0.6282  0.2976  0.2124 

-0.2320  -0.5331  0.6282  1.0000  0.8537  0.8011 

-0.5793  -0.8142  0.2976  0.8537  1.0000  0.9706 

-0.6499  -0.8616  0.2124  0.8011  0.9706  1.0000 


area  =  0.8804  STD.  DEV. (AREA)  =  0.0438 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 


TPF 


(LOWER  BOUND,  UPPER  BOUND) 


0 

.005 

0 

.1840 

( 

0 

.010  - 

0 

.2571 

( 

0 

.020 

0 

.3514 

( 

0 

.030 

0 

.4169 

( 

0 

.040 

0 

.4680 

( 

0 

.050 

0 

.5099 

( 

0 

.060 

0 

.5455 

( 

0 

.070 

0, 

.5765  . 

( 

0, 

.080 

0, 

.6038 

( 

0. 

.090 

0. 

.6282 

( 

0. 

.100 

0, 

.6502 

( 

0. 

.110 

0, 

.6703 

( 

0. 

.120 

0. 

,6886 

( 

0. 

.130 

0. 

.7055 

( 

0. 

.140 

0. 

.7211 

( 

0, 

.150 

■O', 

.7356 

( 

0. 

.200 

0. 

.7949 

( 

0. 

.250 

0. 

.8388 

( 

0, 

,300 

0. 

,8726 

( 

0. 

,400 

0. 

,9205 

( 

0. 

,500 

0. 

,9515 

( 

0. 

,600 

0. 

,9720 

( 

0. 

700 

0. 

9854 

( 

0. 

800 

0. 

9937 

( 

0. 

900 

0. 

9983 

( 

0. 

950 

0. 

9995 

( 

0.0429  ,  0.4670  ) 
0.0914  ,  0.5109  ) 
0.1766  ,  0.5656  ) 
0.2456  ,  0.6060  ) 
0.3013  ,  0.6405  ) 
0.3465  ,  0.6716  ) 
0.3835  ,  0.7002  ) 
0.4141  ,  0.7266  ) 
0.4398  ,  0.7511  ) 
0.4616  ,  0.7736  ) 
0.4803  ,  0.7943  ) 
0.4967  ,  0.8132  ) 
0.5111  ,  0.8305  ) 
0.5240  ,  0.8462  ) 
0.5356  ,  0.8605  ) 
0.5462  ,  0.8736  ) 
0.5882  ,  0.9228  ) 
0.6195  ,  0.9530  ) 
0.6448  ,  0.9717  ) 
0.6859  ,  0.9902  ) 
0.7201  ,  0.9969  ) 
0.7511  ,  0.9992  ) 
0.7815  ,  0.9998  ) 
0.8134  ,  1.0000  ) 
0.8520  ,  1.0000  ) 
0.8792  ,  1.0000  ) 


ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT  LOWER  BOUND  UPPER  BOUND 


(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(0.0032, 

(0.0068, 

(0.0289, 

(0.0903, 

0.1477) 

0.2146) 

0.4108) 

0.6291) 

(0.0002, 

(0.0007, 

(0.0096, 

(0.0461, 

0.0275) 

0.0656) 

0.2524) 

0.4947) 

(0.0325, 

(0.0408, 

(0.0731, 

(0.1602, 

0.4310) 

0.4716) 

0.5855) 

0.7492) 

R  0  C  F  I  T  (JUNE  1993  VERSION)  : 

MAXIMUM  LIKELIHOOD  ESTIMATION 
OF'A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  11,  Mass  Question 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMA 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  100.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  42. 

RESPONSE  DATA : 

CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0600  0.0700  0.2400  1.0000 
TPF:  0.0000  0.6190  0.6667  0.6905  1.0000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  0.6234  B=  0.1714 

Z(K)=  0.7060  1.4761  1.5551 

LOGL=  -113.0300 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  A: 


PROCEDURE  CONVERGES.  AFTER  4  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  0.6617  B=  0.2222 

Z(K)=  0.7099  1.4142  1.5978 

LOGL=  -111.9829 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AI 


VARIANCE-COVARIANCE  MATRIX: 

A  0.0626  0.0207  0.0054  -0.0002  -0.0026 
B  0.0207  0.0174  0.0017  -0.0046  -0.0074 
Z(  1)  0.0054  0.0017  0.0189  0.0132  0.0121 
Z(  2)  -0.0002  -0.0046  0.0132  0.0329  0.0316 
Z(  3)  -0.0026  -0.0074  0.0121  0.0316  0.0415 


CORRELATION  MATRIX: 

A  1.0000  0.6284  0.1570  -0.0048  -0.0516 
B  0.6284  1.0000  0.0938  -0.1911  -0.2756 
Z(  1)  0.1570  0.0938  1.0000  0.5295  0.4318 
Z(  2)  -0.0048  -0.19il  0.5295  1.0000  0.8557 
Z(  3)  -0.0516  -0.2756  0.4318  0.8557  1.0000 


AREA  =  0.7408  STD.  DEV. (AREA)  =  0.0756 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


1  2  3  4  5 

76.  17.  0.  1.  6. 

13.  •  1.  0.  2.  26. 


0.005 

0.5356 

( 

0.3324 

r 

0.7297 

0.010 

0.5575 

( 

0.3684 

9 

0.7342 

0.020 

0.5813 

( 

0.4066 

9 

0.7411 

0.030  ■ 

0.5963 

( 

0.4297 

9 

0.7468 

0.040 

0.6074 

( 

0.4462 

9 

0.7519 

0.050 

.0.6164 

( 

0.4590 

9 

0.7565 

0.060 

0.6240 

( 

0.4694 

9 

0.7608 

0.070 

0.6307 

( 

0.4782 

9 

0.7649 

0.080 

0.6366 

( 

0.4856 

9 

0.7688 

0.090 

0.6420  . 

( 

0.4921 

9 

0.7725 

0.100 

0.6469 

( 

0.4979 

9 

0.7761 

0.110 

0.6514 

( 

0.5030 

9 

0.7796 

0.120 

.  0.6556 

( 

0.5076 

9 

0.7829 

0.130 

0.6596 

( 

0.5117 

9 

0.7862 

0.140 

0.6633 

( 

0.5155 

9 

0.7894 

0.150 

0.6669 

( 

0.5190 

9 

0.7925 

0.200- 

.0.6825 

( 

0.5329 

9 

0.8070 

0.250 

0.6956 

( 

0.5428 

9 

0.8203 

0.300 

0.7072 

( 

0.5503 

9 

0.8325 

0.400 

0.7276 

( 

0.5609 

9 

0.8549 

0.500 

0.7459 

( 

0.5679 

9 

0.8754 

0.600 

0.7636 

( 

0.5729 

9 

0.8947 

0.700 

0.7817 

( 

0.5765 

9 

0.9136 

0.800 

0.8020 

( 

0.5789 

9 

0.9329 

0.900 

0.8280 

( 

0.5800 

9 

0.9546 

0.950 

0.8478 

( 

0.5795 

9 

0.9681 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT  LOWER  BOUND  UPPER  BOUND 

(  FPF  ,  TPF  )  (  FPF  ,  TPF  )  (  FPF  ,  TPF  ) 

(0.0550,  0.6204)  (0.0229,  0.5862)  (0.1154,  0.6537) 

(0.0786,  0.6359)  (0.0384,  0.6058)  (0.1448,  0.6651) 

(0.2389,  0.6928)  (0.1637,  0.6715)  (0.3298,  0.7136) 

R  0  C  F  I  T  (JUNE  1993  VERSION)  : 

MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  11,  MicroCalcifications 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMAI 
NO.  OF  ACTUALLY  NEGATIVE  CASES  =  100.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  51. 

RESPONSE  DATA: 

CATEGORY  1 

ACTUALLY  NEGATIVE  CASES  95. 

ACTUALLY  POSITIVE  CASES  26. 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0400  0.0500  0.0500  1.0000 
TPF:  0.0000  0.4510  0.4510  0.4902  1.0000 

INITIAL  VALUES  OF  PARAMETERS: 

\o 


2  3  4  5 

0.  0.  1.  4. 

2.  0.  0.  23. 


1.7511 


A=  1.5769  B=  0.9713 
Z(K)=  1,5452  1.6452 

LOGL=  -67.9594 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  A 


PROCEDURE  CONVERGES  AFTER  5  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.6025  B=  0.9900 

Z(K)=  1.6444  1.7102  1.7441 

LOGL=  -66.6294 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  A 


A 

VARIANCE-COVARIANCE  MATRIX: 
4.1910  2.4630  0.0735  -0.-0363 

-0.0944 

B 

2.4630 

1.4733 

0.0178 

-0.0482 

-0.0832 

Z( 

1) 

0.0735 

0.0178 

0.0446 

0.0434 

0.0428 

Z( 

2) 

-0.0363 

-0.0482 

0.0434 

0.0474 

0.0483 

Z( 

3) 

-0.0944 

-0.0832 

0.0428 

0.0483 

0.0512 

CORRELATION  MATRIX: 

A 

1.0000  0.9912 

0,1699 

-0.0814 

-0.2037 

B 

0.9912  1.0000 

0.0695 

-0.1825 

-0.3028 

Z( 

1) 

0.1699  0.0695 

1.0000 

0.9449 

0.8960 

Z( 

2) 

-0.0814  -0.1825 

0.9449 

1.0000 

0.9806 

Z( 

3) 

-0.2037  -0.3028 

0.8960  - 

0.9806 

1.0000 

AREA  =  0.8726 

STD 

.  DEV. (AREA)  = 

0.1617 


ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


0.005 

0.010 

0.020 

0.030 

0.040 

0.050 

0.060 

0.070 

0.080 

0.090 

0.100 

0.110 

0.120 

0.130 

0.140 

0.150 

0.200 

0.250 

0.300 

0.400 

0.500 

0.600 

0.700 

0.800 


0.1716 

( 

0.0008 

/ 

0.8977 

) 

0.2416 

( 

0.0095 

r 

0.8277 

) 

0.3332 

( 

0.0688 

f 

0.7331 

) 

0.3975 

( 

0.1615 

r 

0.6802 

) 

0.4478' 

( 

0.2436 

/ 

0.6673 

) 

0.4895 

( 

0.2873 

r 

0.6945 

) 

0.5251 

( 

0.2958 

t 

0.7461 

) 

0.5561 

( 

0.2860 

r 

0.8016 

) 

0.5836 

( 

0.2690 

r 

0.8504 

). 

0.6083 

( 

0.2500 

r 

0.8896 

) 

0.6306 

( 

0.2310 

r 

0.9197 

) 

0.6510 

( 

0.2128 

r 

0.9421 

) 

0.6697 

( 

0.1959 

r 

0.9586 

) 

0.6870 

( 

0.1802 

f 

0.9706 

) 

0.7030 

( 

0.1658 

r 

0.9792 

) 

0.7178 

( 

0.1525 

f 

0.9853 

) 

0.7792 

( 

0.1010 

r 

0.9976 

) 

0.8251 

( 

0.0672 

t 

0.9996 

) 

0.8608 

( 

0.0448 

r 

0.9999 

) 

0.9118 

( 

0.0195 

t 

1.0000 

) 

0.9455 

( 

0.0080 

f 

1.0000 

) 

0.9681* 

( 

0.0029 

f 

1.0000 

) 

0.9831 

( 

0.0009 

f 

1.0000 

) 

0.9926 

( 

0.0002 

r 

1.0000 

) 

0.900 

0.950 


0.9980 

0.9994 


( 

( 


0.0000 

0.0000 


1.0000  ) 

1.0000  ) 


ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 
(  FPF  ,  TPF  ) 

LOWER  BOUND 
(  FPF  ,  TPF  ) 

UPPER 
(  FPF  , 

BOUND 

TPF  ) 

(0.0406, 

(0.0436, 

(0.0500, 

0.4506) 

0.4639). 

0.489-8) 

(0.0143,  0.2866) 
(0.0163,  0.3040) 
(0.0198,  0.3316) 

(0.0967, 

(0.0996, 

(0.1093, 

0.6236) 

0.6299) 

0.6497) 

1  R  0  C  F  I  T  (JUNE  1993  VERSION)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 

OF  A  BINORMAL  ROC  CURVE 

FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  11,  FAS/AD 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMA] 
NO.  OF  ACTUALLY  NEGATIVE  CASES  =  100. 

RESPONSE  DATA: 

CATEGORY  1  2 

ACTUALLY  NEGATIVE  CASES  73.  1. 

ACTUALLY  POSITIVE  CASES  13.  1. 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.1200  0.2600  0.2700  1.0000 
TPF:  0.0000  0.1739  0.3913  0.4348  1.0000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  0.6094  B=  1.3192 
Z(K)=  0.6125  0.6430  1.1751 

LOGL=  -106.1220 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AP' 


NO.  OF  ACTUALLY  POSITIVE  CASES  =  23. 


3  4  5 

0.  14.  12. 

0.  5.  4, 


PROCEDURE  CONVERGES  AFTER  3  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  0.6843  B=  1.3990 

Z(K)=  0.6117  0.6559  1.1707 

LOGL=  -105.9773 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AI 


VARIANCE-COVARIANCE  MATRIX: 


A  0.3212  0.2899 

B  0.2899  0.3677 

Z(  1)  0.0325  0.0120 

Z(  2)  0.0300  0.0086 

Z(  3)  -0.0014  -0.0324 


0.0325  0.0300  -0.0014 
0.0120  0.0086  -0.0324 
0.0180  0.0176  0.0132 
0.0176  0.0181  0.0140 
0.0132  0.0140  0.0263 


CORRELATION  MATRIX: 

A 

1.0000 

0.8435 

0.4273 

0.3936 

-0.0149 

B 

0.8435 

1.0000 

0.1470 

0.1056 

-0.3302 

Z( 

1) 

0.4273 

•  0.1470 

1.0000 

0.9726 

0.6091 

Z( 

2) 

0.3936 

0.1056 

0.9726 

1.0000 

0.6421 

Z( 

3) 

-0.0149 

-0.3302 

0.6091 

0.6421 

1.0000 

AREA  = 

0.6547 

STD 

.  DEV. (AREA)  = 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE -POSITIVE  FRACTION: 


FPF  .. 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.0018 

( 

0.0000 

r 

0.2380 

) 

0.010 

0.0051 

( 

0.0000 

/ 

0.2586 

) 

0.020 

0,0143 

( 

0.0001 

/ 

0.2840 

) 

0.030 

0.0257 

( 

0.0004 

r 

0.3022 

) 

0.040 

0.0387 

( 

0.0011 

r 

0.3174 

) 

0.050 

0.0529 

( 

0.0026 

t 

0.3310 

) 

0.060 

0.0679 

( 

0.0049 

t 

0.3438 

) 

0.070 

0.0837 

( 

0.0084 

r 

0.3561 

) 

0.080 

0.1000 

( 

0.0130 

t 

0.3683 

) 

0.090 

0.1167 

( 

0.0188 

/ 

0.3804 

) 

0.100 

0.1337 

( 

0.0259 

f 

0.3927 

) 

0.110 

0.1511 

( 

0.0341 

r 

0.4053 

) 

0.120 

0.1686 

( 

0.0434 

r 

0.4182 

) 

0.130 

0.1863 

( 

0.0536 

t 

0.4316 

) 

0.140 

0.2041 

( 

0.0646 

f 

0.4456 

) 

0.150 

0.2219 

( 

0.0762 

t 

0.4601 

) 

0.200 

0.3110 

( 

0.1372 

t 

0.5426 

) 

0.250 

0.3979 

-  ( 

0.1919 

t 

0.6380 

) 

0.300 

0.4805 

( 

0.2348 

r 

0.7342 

) 

0.400 

0.6295 

( 

0.2939 

f 

0.8855 

) 

0.500 

0.7531 

( 

0.3348 

r 

0.9637 

) 

0.600 

0.8504 

( 

0.3685 

r 

0.9921 

) 

0.700 

0.9218 

( 

0.4003 

t 

0.9990 

) 

0.800 

0.9687 

( 

0.4343 

f 

0.9999 

) 

0.900 

0.9934 

( 

0.4784 

r 

1.0000 

) 

0.950 

0.9986 

( 

0.5130 

t 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT  LOWER  BOUND  UPPER  BOUND 

(  FPF  ,  TPF  )  (  FPF  ,  TPF  )  (  FPF  ,  TPF  ) 

(0.1209,  0.1702)  (0.0683,  0.0811)  (0.1968,  0.3053) 

(0.2559,  0.4078)  (0.1789,  0.2735)  (0.3475,  0.5540) 

(0.2704,  0.4319)  (0.1908,  0.2948)  (0.3637,  0.5779) 

R  0  C  F  I  T  (JUNE  1993  VERSION)  ; 

MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION;  Reader  11,  Benign  or  Malignant 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMA 
NO.  OF  ACTUALLY  NEGATIVE  CASES  =  100.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  100. 

RESPONSE  DATA: 

CATEGORY  123 

ACTUALLY  NEGATIVE  CASES  56.  31.  13. 

ACTUALLY  POSITIVE  CASES  21.  38.  18. 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0000  0.0000  0.1300  0.4400  1.0000 
TPF:  0.0000  0,0900  0.2300  0.4100  0.7900  1.0000 

INITIAL  VALUES  OF  PARAMETERS : 

A=  0.8157  B=  0.7526 

Z(K)=  0.1507  1.1265  2.4762  2.5762 

LOGL=  -270.4485 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  A 


PROCEDURE  CONVERGES  AFTER  7  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  0.8626  B=  0.7836 

Z(K)=  0.1283  1.2717  2.1074  2.8588 

LOGL=  -249.0090 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  A 


VARIANCE-COVARIANCE  MATRIX : 

A  0.0316  0.0134-  0.0127  0.0070  -0.0015  -0.0119 

B  0.0134  0.0174  0.0044  -0.0078  -0.0229  -0.0396 

Z(  1)  0.0127  0.0044  0,0157  0.0086  0.0045  0.0006 

Z(  2)  0.0070  -0.0078  0.0086  0.0247  0.0289  0.0356 

Z(  3)  -0.0015  -0.0229  0.0045  0.0289  0.0647  0.0826 

Z(  4)  -0.0119  -0.0396  0.0006  0.0356  0.0826  0.1506 


CORRELATION  MATRIX: 

A  1.0000  0.5699  0.5690  0.2500  -0.0329  -0.1725 

B  0.5699  1.0000  0.2638  -0.3768  -0.6815  -0.7725 

Z(  1)  0.5690  0.2638  1.0000  0.4374  0.1427  0.0120 

Z(  2)  0.2500  -0.3768  0.4374  1.0000  0.7221  0.5829 

Z(  3)  -0.0329  -0.6815  0.1427  0.7221  1.0000  0.837,3 

Z(  4)  -0.1725  -0.7725  0.0120  0.5829  0.8373  1.0000 


AREA  =  0.7514  STD.  DEV.  (AREA)  =  0.0382 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.1238 

(  0.0441  , 

0.2718  ) 

0.010 

0.1684 

(  0.0728  , 

0.3207  ) 

0.020 

0.2275 

(  0.1178  , 

0.3791  ) 

0.030 

0.2704 

(  0.1544  , 

■  0.4187  ) 

4  5 

0.  0. 

14.  9. 


0.040 

0.3052 

0.050 

0.3349 

0.060 

0.3609 

0.070  • 

0.3844 

0.080 

0.4057 
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ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
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UNIVERSITY  OF  WASHINGTON 

SEATTLE/WASHINGTON  98195 


Brent  K.  Stewart,  Ph.D. 
Associate  Professor  and  Director 
Diagnostic  Physics  Laboratory 
Department  of  Radiology,  RC-05 


Samuel  J.  Dwyer  HI,  Ph.D. 
Professor 

Department  of  Radiology 
University  of  Virginia 
MR-4  Room  1190 
Charlottesville,  VA  22908 

Dear  Sam: 


NW040J  Health  Sciences  Bldg. 
206.548.6252  (office) 
206.543.3495  (fax) 
bstewart@u.washington.edu 


19  March  1995 


Please  find  enclosed  the  findings  of  my  visit  to  the  Medical  College  of  Virginia  on 
1/27/95  and  the  University  of  Virginia  on  1/28/95  as  consultant  on  the  US  Army 
Medical  Research  and  Development  Command  grant  entitled:  "Evaluation  of  a  Digital 
Telemammography  System:  a  Model  for  a  Regional  System." 

Visit  to  the  Medical  College  of  Virginia 

On  1/27/95, 1  met  with  Ellen  Shaw  de  Parades,  M.D.,  Chief  of  Mammography  at  the 
Medical  College  of  Virginia's  Department  of  Radiology  and  Principal  Investigator  of  the 
telemammography  grant.  The  purpose  of  the  consulting  at  the  Medical  College  of 
Virginia  was  to  analyze  the  design  of  the  Receiver  Operator  Characteristics  (ROC) 
studies,  comment  on  the  method  of  selecting  images  for  the  study,  examine  the  images 
already  collected  for  the  study,  and  discuss  strategies  for  analysis  of  the  ROC  after  the 
testing  has  concluded.  I  prepared  a  list  of  questions  (given  below).  I  also  sat  through  a 
few  of  the  tests  to  give  advice  on  reading  room  environment,  e.g.,  view  box  luminance 
and  glare. 

I  submitted  a  list  of  questions  to  Dr.  de  Parades  regarding  the  analog  film  and  digital 
softcopy  ROC  testing: 

1.  Has  the  ROC  testing  commenced  and  if  so,  how  far  along  is  it? 

2.  Has  the  ROC  study  design  changed  significantly  from  that  stated  in  the  initial 
proposal? 

3.  How  are  the  mammograms  for  the  ROC  study  selected? 

4.  You  are  selecting  age-matched  normal  controls.  Are  you  matching  these  normal 
mammograms  for  overall  parenchymal  density  as  well?  If  so,  how  are  you 
accomplishing  this? 

5.  Are  the  initial  200  mammograms  cited  in  the  grant  application  digitized  yet?  If  so, 
what  is  keeping  you  from  initiating  the  digital  softcopy  ROC  portion  of  the  study? 


6.  For  mammogram  digitization,  what  quality  control /assurance  program  have  you 
instituted? 

7.  In  the  original  grant  application,  it  was  stated  that  the  digital  softcopy  review  might 
occur  on  any  of  ten  different  2K  resolution  workstations  throughout  the  UVA 
Department  of  Radiology.  Unless  these  workstation's  monitors  are  periodically  and 
effectively  calibrated,  this,  might  confound  the  RCXZ  results. 

8.  In  the  original  grant  application,  the  images  read  at  the  remote  Northridge  outpatient 
clinic  were  to  be  subjected  to  a  preference  test  (scale:  1-5).  Would  it  be  better  to  have 
these  cases  overread  by  mammographers  at  UVA  and  statistically  calculate  the  analysis 
of  variance? 

9.  Reading  all  of  the  analog  images  first  produces  a  bias  in  the  ROC  test.  It  would  be 
better  to  have  one-half  of  the  radiologists  read  the  digital  softcopy  images  first  and  the 
other  radiologists  read  the  analog  images  first.  Of  course,  as  there  will  be  multiple 
reading  sessions  for  each  modality,  each  session  could  be  randomly  picked  from  analog 
or  digital.  This  bias  may,  of  course,  be  confounded  as  the  radiologists  will  know  which 
images  are  analog  and  which  are  digital. 

10.  Is  ground  truth  available  for  all  of  the  films  to  be  used  in  the  study?  What  aside 
information  are  you  using  to  establish  ground  truth?  Is  there  a  truth  committee?  If  so, 
who  is  on  it  and  how  do  they  arrive  a  conclusion  regarding  a  case  without  unanimity? 

11.  A  random  number  generator  should  also  be  used  for  ordering  the  analog  and  digital 
normals  and  abnormals  in  each  of  the  ROC  study  sessions.  Is  this  the  case  and  if  not, 
why  not? 

12.  Who  will  be  collecting,  collating  and  performing  the  analysis  of  the  ROC  test  result 
data?  Will  you  be  using  one  of  the  standard  software  packages  like  ROCFIT  or 
CORROC  from  the  University  of  Chicago? 

13.  In  the  grant,  it  is  stated  that  in  addition  to  the  50pm  digitized  radiographs,  that  some 
would  be  digitized  at  23pm.'  If  so,  how  many  and  are  you  adding  this  as  another  section 
of  the  original  ROC  study? 

14.  It  appears  from  the  grant  application  that  the  ROC  results  will  be  pooled  for  the  four 
different  pathology  types.  Is  this  still  the  case?  Will  you  achieve  sufficient  statistical 
power  in  the  non-pooled  case? 

15.  How  has  splitting  the  grant  across  institutions  (UVA  and  MCV)  affected  the  design 
and  execution  of  the  proposed  work? 

16.  Will  you  be  using  the  BIRADS  system  information  for  patient  selection?  It  doesn't 
appear  that  the  RadCare  radiology  information  system  in  place  at  the  MCV  will 
facilitate  on-line  image  selection  for  the  ROC.  What  system  will  you  have  to  help 
automate  patient  selection? 

17.  Are  there  any  problems  in  selecting  cases  from  both  MCV  and  UVA  in  terms  of 
image  quality  differences?  There  should  be  differences  in  film  type,  screen  type, 
mammography  machine  output,  film  processing,  etc. 

18.  It  will  be  possible  to  time  the  readers  using  the  computer  in  the  softcopy  display 
workstation.  Are  you  planning  on  doing  this?  If  so,  could  the  radiologist  write-down 
the  start  and  end  times  on  the  scoring  sheets? 


Advice  on  ROC  Reading  Sessions: 

I  also  sat  through  three  sessions  of  analog  ROC  testing  with  one  private  practice 
mammographer  and  two  MCV  faculty  radiologists.  A  specific  mammography  view 
panel  was  used.  This  viewing  panel  had  the  capability  of  shuttering  out  extraneous 
light  around  the  edges  of  the  films,  however,  this  was  not  done  in  all  cases  by  all 
radiologists.  Both  8"xl0"  and  10"xl2"  films  were  used.  One  row  was  used  at  a  time. 
The  medio-lateral  views  paired  on  the  left,  the  cranial-caudal  radiographs  were  paired 
on  the  right.  The  room,  the  mammography  reading  room,  was  fairly  quiet,  but  was 
simultaneously  used  by  another  radiologist  and  a  resident,  as  well  as  Dr.  de  Parades 
during  the  ROC  Sessions.  The  readers  did  interrupt  their  reading  sessions  to  speak  with 
colleagues  or  answer  the  phone /pager.  There  were  no  overhead  lights  to  contend  with 
and  there  was  no  light  reflections  on  the  ROC  viewbox. 

A  magnifying  lens  was  provided  (will  an  analogous  "zooming"  capability  be  added  to 
the  softcopy  display  workstation  as  well?).  A  hot  lamp  was  available  (will  an  analogous 
grayscale  look-up  table  facility  be  added  to  the  softcopy  display  workstation  as  well?). 

The  reading  sessions  consist  of  50  patient  studies,  each  consisting  of  four  radiographs  (2 
CC/2MLO).  There  is  one  three  ring  binder  notebook  for  each  reader.  All  of  the 
instructions  for  each  reader  are  in  the  notebooks,  as  well  as  all  of  the  reader  responses 
for  each  patient  case  read. 

On  the  average,  the  magnifying  glass  was  used  in  96%  of  the  cases  read,  whereas,  the 
hot  lamp  was  used  only  sparingly,  about  10%  of  the  time.  The  radiologists  always 
started  with  the  MLO  views  and  then  the  CC  views.  Any  zooming  and  panning  would 
need  to  happen  quickly  to  be  effective  (not  slowing  down  the  reading  process 
significantly.  There  were  several  instances  of  the  radiologists  being  interrupted  for 
pages  and  consultations.  If  a  timer  were  to  be  integrated  into  the  softcopy  reading 
workstation,  a  pause  button  would  be  useful. 

There  were  several  instances  where  films  were  displaced  vertically  to  come  into 
registration  (vertical  shift).  This  capability  may  need  to  be  added  to  the  digital  review 
workstation.  It  would  be  very  hard  to  be  the  video  monitors  close  enough  for  bi-lateral 
comparison.  Digital  parming  may  be  necessary.  On  the  average  it  took  two  minutes 
and  18  seconds  to  read  one  of  the  50  studies  in  the  ROC  study  list. 

Visit  to  the  University  of  Virginia 

On  1/28/95, 1  met  with  Samuel  J.  Dwyer,  Ph.D.,  Director  of  PACS  and  Co-Prinicpal 
Investigator  of  the  telemammography  grant  at  the  Medical  College  of  Virginia's 
Department  of  Radiology.  I  also  met  with  Beth  Elias,  B.S.,  the  systems  analyst  for  the 
telemammography  grant.  The  purpose  of  the  consulting  at  the  University  of  Virginia 
was  to  examine  and  provide  recommendations  for  mammogram  digitization,  image 
presentation  on  the  viewing  monitors,  and  image  processing  functionality. 


I  made  several  recommendations  regarding  image  digitization  quality  control, 
specifically  daily  digitization  of  a  standard  test  pattern  and  periodic  cleaning  and 
calibration  of  the  digitizer.  I  also  suggested  several  means  of  displaying  the  image 
digitally  to  the  radiologists  for  that  portion  of  the  ROC  testing.  There  were  also 
questions  regarding  where  an  additional  image  reading  station  for  the  MCV  portion  of 
the  digital  ROC  testing  were  coming  from.  It  might  be  the  case  the  E-systems  will  loan 
as  system  to  the  MCV  for  the  duration  of  the  ROC  testing.  Due  to  construction  and  a 
snow  storm,  it  was  not  convenient  to  visit  the  Northridge  site. 

Image  Digitization: 

The  images  are  being  digitized  at  the  UVA  under  the  direction  of  Ms.  Elias.  A  Lumisys 
digitizer,  model  1-50,  is  being  used  for  the  digitization.  A  SMPTE  (Society  of  Motion 
Picture  Test  Engineers)  is  being  used  for  daily  grayscale  and  resolution  quality  control. 
The  mirrors  of  the  system  are  cleaned  bi-monthly.  Every  four  months,  a  field  engineer 
from  E-systems  recalibrated  the  digitizer  densitometry. 

It  was  suggested  that  the  name  of  the  patient,  the  patient  identification  number,  the  date 
of  the  examination  and  the  name  of  the  institution  be  masked  off  with  electrical  tape 
prior  to  digitization.  It  was  also  suggested  that  a  single  normal  mammogram  be  used 
for  daily  grayscale  quality  control.  This  mammogram  could  be  digitized  every  day, 
prior  to  digitization  of  mammograms  for  the  digital  part  of  the  study.  Once  registered 
spatially,  the  daily  mammogram  could  be  digitally  subtracted  from  the  baseline  one  and 
the  difference  image  studied.  If  it  appeared  that  there  is  more  than  simple  noise 
differences  in  the  difference  image,  e.g.,  structure  evident,  then  the  densitometry  might 
need  to  be  adjusted  more  often  than  every  four  months. 

Image  Presentation  on  the  Viewing  Monitors: 

How  many  monitors  are  going  to  be  used  for  the  workstations  in  the  study?  Only  two. 
It  was  observed  above  that  the  radiologists  reviewing  the  analog  cases  switched  back 
and  forth  between  the  CC  and  MLO  pairs  quite  often.  If  only  two  video  monitors  were 
used,  this  would  severely  hamper  both  the  comparison  necessary  for  diagnosis,  but 
significantly  increase  the  interpretation  time  as  well.  Methods  were  discussed  with  Ms. 
Elias  for  quickly  context  switching  between  the  two  sets  (MLO  and  CC)  of 
mammograms  for  each  patient.  The  limiting  factor  here  is  that  it  is  only  possible  to  load 
two  mammograms  into  the  E-systems  MegaScan  2K  monitor  digital  frame  buffer  (32 
Mbyte  limit).  Having  to  re-paint  the  frame  buffers  from  magnetic  disk  for  each  MLO  <- 
>  CC  context  switch  will  most  likely  be  interminably  slow. 

It  was  also  suggested  that  a  sequential  worklist  of  patients  for  the  softcopy  review 
workstation  portion  of  the  ROC  study  be  instituted.  Currently,  the  radiologist  has  to 
select  images  from  a  pull-down  menu  list  with  small  font.  The  easiest  thing  for  the 
radiologists  to  have  to  do  would  be  to  push  a  "hot  key"  to  advance  to  the  next  patient  in 
the  ROC  study  list  automatically.  Otherwise,  with  the  limitations  of  the  MLO  <->  CC 
context  switching  and  having  to  search  through  a  complicated  list  of  code  numbers,  the 
radiologists  will  become  frustrated,  which  might  impact  the  results  of  that  portion  of 
the  ROC  test. 


Image  Processing: 

In  order  to  emulate  the  functionality  of  the  hot  lamp  and  the  magnifying  glass,  image 
processing  functions  will  be  implemented  on  the  digital  viewing  station.  However,  the 
zooming  functionality  included  with  the  E-systems  MegaScan  monitors  looks  overly 
complicated  for  a  function  that  the  radiologists  used  about  96%  of  the  time  in  the  analog 
portions  of  the  ROC  tests. 

With  regards  to  grayscale  modifications  of  the  digital  mammograms,  the  user  can 
change  both  the  brightness  and  the  contrast.  This  is  accomplished  fairly  easily  using  the 
mouse,  moving  it  either  up  or  down  for  contrast  modification  and  left  to  right  for 
brightness/ darkness  changes.  However,.as  there  are  three  buttons  on  the  optical 
mouse,  a  specific  series  of  button  pushes  are  necessary  to  invoke  and  dismiss  the 
grayscale  look-up  table  modification  software.  The  radiologists  are  going  to  have  to 
have  something  simple  to  get  through  the  set  of  50  image  cases  in  a  reasonable  amoimt 
of  time.  I  can  foresee  a  great  amount  of  frustration  with  the  current  user  interface  for 
zooming  and  look-up  table  modification.  All  but  one  of  the  mouse  buttons  should  be 
disabled  for  the  ROC  testing. 


Please  let  me  know  if  there  is  anything  else  that  you  may  require  in  this  matter.  It  has 
been  a  pleasure  working  with  you  and  Dr.  de  Parades  on  the  telemammography 
project. 

Sincerely, 

Brent  K.  Stewart,  Ph.D. 

Consultant  to  the  US  Army  Medical  Research  and  Development  Command  Grant 
Evaluation  of  a  Digital  Telemammography  System:  a  Model  for  a  Regional  System 


APPENDIX  3 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient: 
Patient  ID: 
Home  Phone  # 
DOB: 

Exam  Date: 


Jane  Doe 
020 

09/23/1927 

08/08/1996 


Mammogram  Findings 

There  are  scattered  f ibroglandular  densities  that  could 
obscure  a  lesion  on  mammography.  . 

There  is  an  isodense,  irregular  mass  measuring  12 
millimeters  with  spiculated  margins  seen  in  the  right 
breast  at  2  o'clock. 

Impression 

Mass  in  the  right  breast  is  highly  suggestive  of 
malignancy.  Biopsy  should  be  considered. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patients 
Patient  ID; 
Home  Phone  #: 
DOB: 

Exam  Date: 


Jane  Doe 
079 

01/01/1941 

08/01/1996 


Mammogram  Findings 

There  are  scattered  f ibroglandular  densities  that  could 
obscure  a  lesion  on  mammography.  . 

There  is  an  isodense,  oval  mass  measuring  10 
millimeters  with  circumscribed  margins  seen  in  the  left 
breast  at  12  o'clock. 

Impression 

Mass  in  the  left  breast  is  suspicious.  Biopsy  should 
be  considered. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient: 
Patient  ID: 
Home  Phone  # 
DOB: 

Exam  Date: 


Jane  Doe 
144 

01/01/1943 

08/08/1996 


Mammogram  Findings 

The  breasts  are  heterogeneously  dense.  This  may  lower 
the  sensitivity  of  mammography. 

There  are  amorphous  calcifications  with  grouped 
distribution  seen  in  the  right  breast  at  10  o'clock. 

Impression 

Calcifications  in  the  right  breast  are  suspicious. 
Biopsy  should  be  considered. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient: 
Patient  ID: 
Home  Phone  #: 
DOB: 

Exam  Date: 


Jane  Doe 
143 

03/20/1938 

08/08/1996 


Mammogram  Findings 

The  breasts  are  heterogeneously  dense.  This  may  lower 
the  sensitivity  of  mammography. 

There  is  an  isodense,  round  mass  measuring  18 
millimeters  with  indistinct  margins  seen  in  the 
axillary  tail  of  the  left  breast. 

Impression 

Mass  in  the  left  breast  is  highly  suggestive  of 
malignancy.  Biopsy  should  be  considered. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient: 
Patient  ID: 
Home  Phone  # 
DOB: 

Exam  Date: 


Jane  Doe 
133 

10/28/1935 

08/08/1996 


Mammogram  Findings 

The  breasts  are  heterogeneously  dense.  This  may  lower 
the  sensitivity  of  mammography. 

There  are  heterogeneous  calcifications  with  grouped 
distribution  seen  in  the  left  breast  at  6  o'clock. 

Impression 

Calcifications  in  the  left  breast  are  suspicious. 
Biopsy  should  be  considered. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient: 
Patient  ID: 
Home  Phone  # 
DOB : 

Exam  Date: 


Jane  Doe 
135 

06/29/1927 

08/08/1996 


Mammogram  Findings 

There  are  scattered  f ibroglandular  densities  that  could 
obscure  a  lesion  on  mammography. . 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient; 
Patient  ID: 
Home  Phone  # 
DOB: 

Exam  Date: 


Jane  Doe 
136 

06/01/1936 

08/08/1996 


Mammogram  Findings 

The  breasts  are  heterogeneously  dense.  This  may  lower 
the  sensitivity  of  mammography.  . 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient: 
Patient  ID: 
Home  Phone  #: 
DOB: 

Exam  Date: 


Jane  Doe 
137 

12/01/1986 

08/08/1996 


Mammogram  Findings 

There  are  scattered  f ibroglandular  densities  that  could 
obscure  a  lesion  on  mammography. . 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient : 
Patient  ID: 
Home  Phone  #i 
DOB: 

Exam  Date: 


Jane  Doe 
139 

11/18/1920 

08/08/1996 


Mammogram  Findings 

The  breasts  are  almost  entirely  fat. 

There  is  an  isodense,  round  mass  measuring  5 
millimeters  with  indistinct  margins  seen  in  the  central 
region  of  the  right  breast. 

Impression 

Mass  in  the  right  breast  is  suspicious.  Biopsy  should 
be  considered. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient; 
Patient  ID; 
Home  Phone  #; 
DOB; 

Exam  Date; 


Jane  Doe 
132 

01/01/1957 

08/08/1996 


Mammogram  Findings 

The  breasts  are  heterogeneously  dense.  This  may  lower 
the  sensitivity  of  mammography. 

There  are  amorphous  calcifications  with  grouped 
distribution  seen  in  the  right  breast  at  10  o'clock. 

Impression 

Calcifications  in  the  right  breast  are  suspicious. 
Biopsy  should  be  considered. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient: 
Patient  ID: 
Home  Phone  #: 
DOB: 

Exam  Date: 


Jane  Doe 
130 

03/18/1913 

08/08/1996 


Mammogram  Findings 

There  are  scattered  f ibroglandular  densities  that  could 
obscure  a  lesion  on  mammography. . 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient i 
Patient  ID; 
Home  Phone  # 
DOB: 

Exam  Date: 


Jane  Doe 
145 

12/30/1937 

08/08/1996 


Mammogram  Findings 

The  breasts  are  extremely  dense,  which  lowers  the 
sensitivity  of  mammography. 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient:. 
Patient  ID: 
Home  Phone  # 
DOB: 

Exam  Date: 


Jane  Doe 
146 

04/09/1937 

08/08/1996 


Mammogram  Findings 

The  breasts  are  almost  entirely  fat. 

There  is  a  high  density,  round  mass  measuring  12 
millimeters  with  circumscribed  margins  seen  in  the 
right  breast  at  2  o'clock. 

Impression 

Mass  in  the  right  breast  is  suspicious.  Biopsy  should 
be  considered. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient: 
Patient:  ID: 
Home  Phone  # 
DOB: 

Exam  Date: 


Jane  Doe 
147 

08/02/1924 

08/08/1996 


Mammogram  Findings 

The  breasts  are  almost  entirely  fat. 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient: 
Patient  ID: 
Home  Phone  # 
DOB: 

Exam  Date: 


Jane  Doe 
148 

09/10/1938 

08/08/1996 


Mammogram  Findings 

There  are  scattered  f ibroglandular  densities  that  could 
obscure  a  lesion  on  mammography. 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient: 
Patient  ID: 
Home  Phone  #: 
DOB: 

Exam  Date: 


Jane  Doe 
149 

10/24/1942 

08/08/1996 


Mammogram  Findings 

The  breasts  are  extremely  dense,  which  lowers  the 
sensitivity  of  mammography. 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.O.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient: 
Patient  ID: 
Home  Phone  # 
DOB: 

Exam  Date: 


Jane  Doe 
150 

09/21/1922 

08/08/1996 


Mammogram  Findings 

There  are  scattered  f ibroglandular  densities  that  could 
obscure  a  lesion  on  mammography. 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient,;  Jane  Doe 

Patient  ID:  126 

Home  Phone  #: 
dob:  06/17/1925 

Exam  Date:  08/08/1996 


Mammogram  Findings 

There  are  scattered  f ibroglandular  densities  that  could 
obscure  a  lesion  on  mammography. 

There  is  a  tubular  density  measuring  20  millimeters 
seen  in  the  subareolar  region  of  the  left  breast. 

Impression 

Tubular  density  in  the  left  breast  is  suspicious. 

Biopsy  should  be  considered. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient; 
Patient  ID: 
Home  Phone  # 
DOB: 

Exam  Date: 


Jane  Doe 
017 

01/01/1944 

08/08/1996 


Mammogram  Findings 

There  are  scattered  f ibroglandular  densities  that  could 
obscure  a  lesion  on  mammography. 

There  is  an  isodense,  round  mass  measuring  7 
millimeters  with  circumscribed  margins  seen  in  the 
posterior  region  of  the  left  breast  at  6  o'clock. 

Impression 

Mass  in  the  left  breast  is  suspicious.  Biopsy  should 
be  considered. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient  r 
Patient  ID: 
Home  Phone  # 
DOB: 

Exam  Date: 


Jane  Doe 
018 

06/18/1943 

08/08/1996 


Mammogram  Findings 

The  breasts  are  heterogeneously  dense.  This  may  lower 
the  sensitivity  of  mammography. 

There  are  amorphous  calcifications  with  grouped 
distribution  seen  in  the  right  breast  at  12  o'clock. 

Impression 

Calcifications  in  the  right  breast  are  suspicious. 
Biopsy  should  be  considered. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient: 
Patient  ID: 
Home  Phone  #: 
DOB: 

Exam  Date: 


Jane  Doe 
023 

01/01/1960 

08/08/1996 


Mammogram  Findings 

The  breasts  are  heterogeneously  dense.  This  may  lower 
the  sensitivity  of  mammography. 

There  is  an  isodense,  oval  mass  measuring  12 
millimeters  with  obscured  margins  seen  in  the  left 
breast  at  12  o'clock. 

Impression 

Mass  in  the  left  breast  is  suspicious.  Biopsy  should 
be  considered. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient : 
Patient  ID: 
Home  Phone  # 
DOB: 

Exam  Date: 


Jane  Doe 
049 

12/17/1915 

08/08/1996 


Mammogram  Findings 

The  breasts  are  heterogeneously  dense.  This  may  lower 
the  sensitivity  of  mammography. 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogrami  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient: 
Patient  ID: 
Home  Phone  # 
DOB: 

Exam  Date: 


Jane  Doe 
016 

06/07/1944 

08/08/1996 


Mammogram  Findings 

The  breasts  are  extremely  dense,  which  lowers  the 
sensitivity  of  mammography. 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D 


Consultation  Report 
Nelson  Clinic  Maimography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient : 
Patient  ID: 
Home  Phone  #: 
DOB: 

Exam  Date: 


Jane  Doe 
016 

06/07/1944 

08/08/1996 


Mammogram  Findings 

The  breasts  are  extremely  dense,  which  lowers  the 
sensitivity  of  mammography. 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient: 
Patient  ID: 
Home  Phone  # 
DOB: 

Exam  Date: 


Jane  Doe 
048 

12/11/1922 

08/08/1996 


Mammogram  Findings 

There  are  scattered  f ibroglandular  densities  that  could 
obscure  a  lesion  on  mammography. . 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient: 
Patient  ID: 
Home  Phone  #: 
DOB: 

Exam  Date: 


Jane  Doe 
042 

04/25/1934 

08/08/1996 


Mammogram  Findings 

There  are  scattered  f ibroglandular  densities  that  could 
obscure  a  lesion  on  mammography. . 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Manumography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient: 
Patient  ID: 
Home  Phone  # 
DOB: 

Exam  Date: 


Jane  Doe 
001 

03/23/1949 

08/08/1996 


Mammogram  Findings 

The  breasts  are  heterogeneously  dense.  This  may  lower 
the  sensitivity  of  mammography. 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient: 
Patient  ID: 
Home  Phone  # 
DOB: 

Exam  Date: 


Jane  Doe 
010 

04/26/1934 

08/08/1996 


Mammogram  Findings 

The  breasts  are  almost  entirely  fat. 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient: 
Patient  ID: 
Home  phone  # 
DOB: 

Exam  Date: 


Jane  Doe 
038 

01/27/1948 

08/08/1996 


Mammogram  Findings 

The  breasts  are  heterogeneously  dense.  This  may  lower 
the  sensitivity  of  mammography. 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient; 
Patient  ID: 
Home  Phone  #: 
DOB: 

Exam  Date: 


Jane  Doe 
034 

01/19/1939 

08/08/1996 


Mammogram  Findings 

The  breasts  are  almost  entirely  fat. 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient: 
Patient-  ID: 
Home  Phone  # 
DOB: 

Exam  Date: 


Jane  Doe 
009 

04/20/1921 

08/08/1996 


Mammogram  Findings 

The  breasts  are  almost  entirely  fat. 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient; 
Patient  ID: 
Home  Phone  # 
DOB; 

Exam  Date; 


Jane  Doe 
046 

08/07/1943 

08/08/1996 


Mammogram  Findings 

The  breasts  are  extremely  dense,  which  lowers  the 
sensitivity  of  mammography. 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient: 
Patient  ID: 
Home  Phone  # 
DOB: 

Exam  Date: 


Jane  Doe 
015 

06/08/1942 

08/08/1996 


Mammogram  Findings 

There  are  scattered  f ibroglandular  densities  that  could 
obscure  a  lesion  on  mammography.  . 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient: 
Patient  ID: 
Home  Phone  #: 
DOB: 

Exam  Date: 


Jane  Doe 
047 

12/11/1912 

08/08/1996 


Mammogram  Findings 

There  are  scattered  f ibroglandular  densities  that  could 
obscure  a  lesion  on  mammography. . 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient: 
Patient  ID: 
Home  Phone  # 
DOB: 

Exam  Date: 


Jane  Doe 
026 

07/06/1917 

08/08/1996 


Mammogram  Findings 

The  breasts  are  almost  entirely  fat. 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient; 
Patient  ID : 
Home  Phone  # 
DOB: 

Exam  Date: 


Jane  Doe 
043 

09/06/1945 

08/08/1996 


Mammogram  Findings 

The  breasts  are  heterogeneously  dense.  This  may  lower 
the  sensitivity  of  mammography. 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient: 
Patient  ID: 
Home  Phone  #: 
DOB: 

Exam  Date: 


Jane  Doe 
002 

05/04/1944 

08/08/1996 


Mammogram  Findings 

There  are  scattered  f ibroglandular  densities  that  could 
obscure  a  lesion  on  mammography.  . 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient: 
Patient  ID: 
Home  Phone  # 
DOB: 

Exam  Date: 


Jane  Doe 
044 

10/19/1934 

08/08/1996 


Mammogram  Findings 

There  are  scattered  f ibroglandular  densities  that  could 
obscure  a  lesion  on  mammography.  . 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient: 
Patient  ID: 
Home  Phone  #: 
DOB: 

Exam  Date: 


Jane  Doe 
025 

08/04/1920 

08/08/1996 


Mammogram  Findings 

There  are  scattered  f ibroglandular  densities  that  could 
obscure  a  lesion  on  mammography.  . 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Maitmiography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient: 
Patient  ID : 
Home  Phone  # 
DOB: 

Exam  Date: 


Jane  Doe 
035 

01/16/1945 

08/08/1996 


Mammogram  Findings 

The  breasts  are  extremely  dense,  which  lowers  the 
sensitivity  of  mammography. 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient: 
Patient  ID: 
Home  Phone  #: 
DOB: 

Exam  Date: 


Jane  Doe 
022 

01/27/1952 

08/08/1996 


Mammogram  Findings 

The  breasts  are  heterogeneously  dense.  This  may  lower 
the  sensitivity  of  mammography. 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient: 
Patient  ID: 
Home  Phone  # 
DOB: 

Exam  Date: 


Jane  Doe 
021 

02/02/1939 

08/08/1996 


Mammogram  Findings 

There  are  scattered  f ibroglandular  densities  that  could 
obscure  a  lesion  on  mammography.  . 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient; 
Patient  ID: 
Home  Phone  # 
DOB; 

Exam  Date: 


Jane  Doe 
Oil 

01/01/1900 

08/08/1996 


Mammogram  Findings 

The  breasts  are  almost  entirely  fat. 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M. D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient:  Jane  Doe 

Patient,  ID:  028 

Home  Phone  #: 

DOB:  01/01/1900 

Exam  Date:  08/08/1996 


Mammogram  Findings 

The  breasts  are  almost  entirely  fat. 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient: 
Patient  ID: 
Home  Phone  #: 
DOB: 

Exam  Date: 


Jane  Doe 
031 

08/01/1931 

08/08/1996 


Mammogram  Findings 

The  breasts  are  almost  entirely  fat. 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Manunography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient; 
Patient  ID: 
Home  Phone  # 
DOB: 

Exam  Date: 


Jane  Doe 
050 

11/12/1932 

08/08/1996 


Mammogram  Findings 

There  are  scattered  f ibroglandular  densities  that  could 
obscure  a  lesion  on  mammography.  . 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient; 
Patient  ID; 
Home  Phone  #; 
DOB; 

Exam  Date; 


Jane  Doe 
127 

09/23/1925 

08/08/1996 


Mammogram  Findings 

The  breasts  are  heterogeneously  dense.  This  may  lower 
the  sensitivity  of  mammography.  . 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


PHYSICIANS 

THE  PARK  AT 
STONY  POINT 


Virginia  Commonwealth  University 


Radiology 

9000  Stony  Point  Parkway 
Richmond,  Virginia  23235 
804  560-8906 
Fax  804  560-7345 


Consultation  Report 
Nelson  Clinic  Mammography 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Patient: 
Patient  ID: 
Home  Phone  #: 
DOB: 

Exam  Date: 


Jane  Doe 
140 

07/14/1928 

08/08/1996 


Mammogram  Findings 

The  breasts  are  heterogeneously  dense.  This  may  lower 
the  sensitivity  of  mammography. 

No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 

Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


PHYSICIANS 

THE  PARK AT 
STONY  POINT 


Consultation  Report 
Nelson  Clinic  Mammography 


Virginia  Commonwealtii  University 


Paredes,  M.D.,  Ellen 
Mcv  Hospitals 
Radiology  Box  980615 
Richmond,  VA  23298 


Radiology 

9000  Stony  Point  Parkway 
Richmond,  Virginia  23235 
804  560-8906 
Fax  804  560-7345 


Patient: 
Patient  ID: 
Home  Phone  # 
DOB: 

Exam  Date: 


Jane  Doe 
]141 

12/16/1928 

08/08/1996 


Mammogram  Findings 

There  are  scattered  f ibroglandular  densities  that  could 
obscure  a  lesion  on  mammography. . 


No  masses,  significant  calcifications  or  other 
abnormalities  are  seen. 


Impression 

There  is  no  mammographic  evidence  of  malignancy. 
Screening  mammogram  in  1  year  is  recommended. 


Thank  you  for  referring  this  patient. 


E.  Michelle  Garris,  M.D. 


